Solid-state imaging device for enlargement of dynamic range

ABSTRACT

An inter-line CCD (IT-CCD) capable of reading signals in two read modes, a field read mode and a frame read mode, is used to control, by a system control means, a CCD exposure and a signal read mode, a short-time exposure signal (Short signal) acquires an image by a field read and a long-time exposure signal (Long signal) acquires an image by a frame read, and these two images are synthesized by a signal synthesizing means to expand a dynamic range.

TECHNICAL FIELD

The present invention relates to a solid-state imaging device capable ofexpanding the dynamic range of picked up images.

BACKGROUND ART

Heretofore, a solid-state imaging device for synthesizing two imagesignals having the amount of exposure different from each other toobtain an image signal with a wide dynamic range is disclosed in, forexample, the gazette of Japanese Laid-Open Patent No. HEI9-214829 andthe gazette of Japanese Laid-Open Patent No. HEI9-275527.

The gazette of Japanese Laid-Open Patent No. HEI9-214829 discloses adigital still camera capable of obtaining an image signal with a widedynamic range by level shifting two continuous field images picked up bythe change of the exposure time, and then synthesizing them into oneframe image.

The gazette of Japanese Laid-Open Patent No. HEI9-275527 discloses adigital still camera capable of obtaining an image signal with a widedynamic range by level shifting a plurality of frame images having adifferent exposure times obtained from a plurality of CCDs, and thensynthesizing them into one frame image.

In addition, there has been known an example of a video camera whichexpands the dynamic range by using a special CCD capable of reading along-time exposure signal and a short-time exposure signal within onefield period (“Development of method of processing single-plate Hyper-Dcolor camera signal,” Image Media Society Technical Report, Vol. 22, No.3, pp. 1-6 (1998)).

However, for example, in the digital still camera disclosed in thegazette of Japanese Laid-Open Patent No. HEI9-214829, two continuousfield images picked up by the change of the exposure time aresynthesized, so that the image after synthesized only has an imageresolution for one field, that is, a resolution of a half of the numberof pixels of CCD, whereby a short in the resolution of picked up imageis concerned about.

On the other hand, in the digital still camera disclosed in the gazetteof Japanese Laid-Open Patent No. HEI9-275527, image signals having adifferent exposure times picked up by a plurality of CCDs aresynthesized, so that the image after synthesized has an image resolutionfor one frame, that is, a resolution for the number of pixels of CCD,and however, a plurality of CCDs are required, which is disadvantageousin the size/cost of the imaging device.

For the imaging device having been reported in “Development of method ofprocessing single-plate Hyper-D color camera signal,” Image MediaSociety Technical Report, Vol. 22, No. 3, pp. 1-6 (1998), a special CCDis required to expand the dynamic range of picked up images.

It is an object of the present invention to provide a solid-stateimaging device capable of picking up an image with an expanded dynamicrange at a lower cost and at an image resolution equivalent to thenumber of pixels of CCD by using one CCD used generally with such asolid-state imaging device regardless of consumer or businessapplications.

DISCLOSURE OF THE INVENTION

The solid-state imaging device of the present invention has asolid-state imaging element for outputting a plurality of image signalshaving different exposure times, and signal synthesizer means forsynthesizing image signals outputted from the above-mentionedsolid-state imaging element; at least one of image signals outputtedfrom the above-mentioned solid-state imaging element by the signalsynthesizer means is processed as an image signal having a small numberof pixels than other image signals, thereby allowing an image with anexpanded dynamic range to be picked up at an image resolution equivalentto the number of pixels of CCD.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a solid-state imaging device accordingto Embodiment 1 of the present invention.

FIG. 2 is an illustrative view of a mode of reading signal from thesolid-state imaging device in Embodiment 1 of the present invention.

FIG. 3 is a view showing an example of the arrangement of color filtersformed on the solid-state imaging device in Embodiment 1 of the presentinvention.

FIG. 4 is a block diagram showing a configuration of signal synthesizermeans in Embodiment 1 of the present invention.

FIG. 5 is a block diagram showing a configuration of two horizontal lineadder means in Embodiment 1 of the present invention.

FIG. 6 is a block diagram showing a configuration of interpolation meansin Embodiment 1 of the present invention.

FIG. 7 is a block diagram showing a configuration of weighting addermeans in Embodiment 1 of the present invention.

FIG. 8 is an illustrative view to explain the principle of the dynamicrange expansion in Embodiment 1 of the present invention.

FIG. 9 is an illustrative view to explain the timing of the exposure andread of the long signal and the short signal in Embodiment 1 of thepresent invention.

FIG. 10 is an illustrative view to explain the short signal inEmbodiment 1 of the present invention.

FIG. 11 is an illustrative view to explain the long signal in Embodiment1 of the present invention.

FIG. 12 is an illustrative view to explain the two horizontal lineaddition processing and the interpolation processing in Embodiment 1 ofthe present invention.

FIG. 13 is a graph to explain a synthesis coefficient deciding method inEmbodiment 1 of the present invention.

FIG. 14 is an illustrative view to explain a signal synthesis processingmethod in Embodiment 1 of the present invention.

FIG. 15 is a block diagram showing a configuration of the weightingadder means in Embodiment 2 of the present invention.

FIG. 16 is a block diagram showing a configuration of brightness signalextraction means in Embodiment 2 of the present invention.

FIG. 17 is an illustrative view to explain a long brightness signalproducing method in Embodiment 2 of the present invention.

FIG. 18 is a graph to explain a synthesis coefficient deciding method inEmbodiment 2 of the present invention.

FIG. 19 is an illustrative view to explain a signal synthesis processingmethod in Embodiment 2 of the present invention.

FIG. 20 is a block diagram showing a solid-state imaging device inEmbodiment 3 of the present invention.

FIG. 21 is a block diagram showing a configuration of a brightnesssignal interpolation means in Embodiment 3 of the present invention.

FIG. 22 is a block diagram showing a configuration of brightness signalsynthesizer means in Embodiment 3 of the present invention.

FIG. 23 is a block diagram showing a configuration of signal synthesizermeans in Embodiment 3 of the present invention.

FIG. 24 is a block diagram showing a configuration of synchronizationmeans in Embodiment 3 of the present invention.

FIG. 25 is an illustrative view to explain a long brightness signal inEmbodiment 3 of the present invention.

FIG. 26 is an illustrative view to explain a short brightness signal inEmbodiment 3 of the present invention.

FIG. 27 is an illustrative view to explain a brightness signalinterpolation processing in Embodiment 3 of the present invention.

FIG. 28 is an illustrative view to explain a brightness signalsynthesizing method in Embodiment 3 of the present invention.

FIG. 29 is an illustrative view to explain a synthesization processingby the synthesizer means in Embodiment 3 of the present invention.

FIG. 30 is a block diagram showing a solid-state imaging device inEmbodiment 4 of the present invention.

FIG. 31 is a block diagram showing a configuration of synchronizationmeans in Embodiment 4 of the present invention.

FIG. 32 is an illustrative view to explain a synthesization processingby the synthesizer means in Embodiment 4 of the present invention.

FIG. 33 is an illustrative view showing another example of a method ofreading an image signal from a solid-state imaging element.

FIG. 34 is an illustrative view to explain the two horizontal lineaddition processing and the interpolation processing when taking thelong signal as a field image and the short signal as a frame image inEmbodiment 1 of the present invention.

FIG. 35 is an illustrative view to explain previous-value interpolationprocessing.

FIG. 36 is an illustrative view showing another example of a method ofsynthesizing the long signal and the short signal in Embodiment 1 of thepresent invention.

FIG. 37 is a graph showing another example of a method of deciding asynthesis coefficient from the long signal in Embodiment 1 of thepresent invention.

FIG. 38 is an illustrative view showing another example of a method ofsynthesizing the long signal and the short signal in Embodiment 2 of thepresent invention.

FIG. 39 is a graph showing another example of a method of deciding asynthesis coefficient from the long brightness signal in Embodiment 2 ofthe present invention.

FIG. 40 is a block diagram of the solid-state imaging device whenchanging a short signal reading method in Embodiment 3 of the presentinvention.

FIG. 41 is an illustrative view to explain the contents of thebrightness signal interpolation processing when changing a short signalreading method in Embodiment 3 of the present invention.

FIG. 42 is a block diagram showing another configuration of the signalsynthesizer means in Embodiment 3 of the present invention.

FIG. 43 is an illustrative view showing another example of a method ofsynthesizing the long brightness signal and the short brightness signalin Embodiments 3 and 4 of the present invention.

FIG. 44 is a block diagram showing another example of the solid-stateimaging device in an fourth embodiment of the present invention.

FIG. 45 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element.

FIG. 46 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of CyYeG stripemethod).

FIG. 47 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of bayer method).

FIG. 48 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of inter-linemethod).

FIG. 49 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of G stripe RBcompletely checkered method).

FIG. 50 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of stripe method).

FIG. 51 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of diagonal stripemethod).

FIG. 52 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of G stripe RB linesequential method).

FIG. 53 is a view showing another example of the arrangement of colorfilters formed on the solid-state imaging element (of G stripe RB pointsequential method).

FIG. 54 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 5 of the present invention.

FIG. 55 is a block diagram showing a configuration example of twohorizontal line adder means in the solid-state imaging device inEmbodiment 5 of the present invention.

FIG. 56 is a block diagram showing a configuration example of an imagememory in the solid-state imaging device in Embodiment 5 of the presentinvention.

FIG. 57 is a timing chart with respect to the exposure of a subjectimage, to the read of an exposed signal, and to the read/write operationof the image memory in the solid-state imaging device in Embodiment 5 ofthe present invention.

FIG. 58 is a block diagram showing a configuration example of signallevel determination means in the solid-state imaging device inEmbodiment 5 of the present invention.

FIG. 59 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 6 of the present invention.

FIG. 60 is a timing chart with respect to the exposure of a subjectimage, to the read of an exposed signal, and to the read/write operationof the image memory in the solid-state imaging device in Embodiment 6 ofthe present invention.

FIG. 61 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 7 of the present invention.

FIG. 62 is a typical view showing a state of the screen block divisionin Embodiment 7 of the present invention.

FIG. 63 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 8 of the present invention.

FIG. 64 is a configuration view showing a color filter arrangement ofthe solid-state imaging element in the solid-state imaging device inEmbodiment 8 of the present invention.

FIG. 65 is a graph of the fluctuation in brightness for each colorcomponent of a fluorescent lamp.

BEST MODE FOR CARRYING OUT THE INVENTION Embodiment 1

FIG. 1 is a block diagram of a solid-state imaging device in a best mode(hereinafter called Embodiment 1) in order to carry out the presentinvention. In the diagram, the numeral 1 designates an optical lens; thenumeral 2, a mechanical shutter as light shutter means also used for anoptical aperture; and the numeral 3, a solid-state imaging device, thesolid-state imaging device being an inter line transfer CCD (IT-CCD)generally used with a consumer solid-state imaging device. Other thanthe CCD, there are a frame inter-line transfer CCD (FIT-CCD) and a CMOSsensor. The numeral 4 designates an analog signal processing meansconsisting of a co-related double sampling circuit and of an automaticgain control (AGC) circuit; the numeral 5, an A/D converter means; thenumeral 6, an image memory for storing an image signal converted to adigital signal by the A/D converter means 5; and the numeral 7, signalsynthesizer means for synthesizing two-system image signals read fromthe image memory 6.

A signal obtained by the signal synthesizer means 7 is subject toprocessings such as separating of brightness signal from color signal,noise removing, edge enhancing, matrix operating, and encoding to aspecific format in a digital signal processing means 8. A mechanicalshutter drive control means 9 controls the opening or closing of themechanical shutter 2; and a solid-state imaging element drive controlmeans 10 controls the exposure control, signal read mode and timing ofthe solid-state imaging element 3. The operation mode and operationtiming of all of the above-mentioned components including these meansshould be controlled integrally by a system control means 11.

FIGS. 2( a), 2(b), 2(c) and 2(d) are typical views to explain theoperation/configuration of the solid-state imaging element 3. Thesolid-state imaging element 3 is an inter-line transfer CCD (IT-CCD)capable of reading a signal in two read modes of a field read mode and aframe read mode, which for the sake of explanation, will be explainedwith a configuration consisting of four vertical pixels and twohorizontal pixels, the so-that 4×2 pixels arranged in the twodimensional plane such as in matrix shape as shown in FIG. 2.

FIGS. 2( a) and 2(b) are views to explain the field read mode in theIT-CCD. In FIG. 2( a), a photodiode as an element for storing a chargecorresponding to the amount of an incident light is a portion in which asignal charge according to light intensity by photoelectric conversionis stored, and after a certain time, the accumulated charge istransferred to a vertical transfer CCD by an applied control pulse. Atthis time, charges of two upper/lower photodiodes adjacent to each otherare mixed on the vertical transfer CCD, and outputted through ahorizontal transfer CCD to the outside. The above-mentioned operation isthe read operation of a first field.

In a second field, as shown in FIG. 2( b), a pair of photodiodes mixedon the vertical transfer CCD are dislocated by one pixel in the verticaldirection compared to that for the first field. Thus, the signal readfor two fields allows said image signal equivalent to one frame of theinterlace method to be read.

Using FIGS. 2( c) and 2(d), the frame read mode will be explainedhereinafter. In the frame read mode, first in the first field (FIG. 2(c)), the charge accumulated in the photodiode by one pixel jumping inthe vertical direction is transferred to the vertical transfer CCD, andthen outputted through the horizontal transfer CCD to the outside. Inthe second field (FIG. 2( d)), the charge of the photodiode nottransferred to the vertical transfer CCD in the first field istransferred to the vertical transfer CCD, and then outputted through thehorizontal transfer CCD to the outside. In this manner, in the frameread mode, the charges on the photodiode are not mixed in the verticaltransfer CCD and outputted to the outside. Thus, the signal read for twofields allows said image signal equivalent to one frame of the interlacemethod to be read.

FIG. 3 is an arrangement view of complementary-color checkered typecolor filters formed on the solid-state imaging element 3. In FIG. 3, Mgrepresents magenta color; G, green color; Ye, yellow color; and Cy, cyancolor. As shown in FIG. 3, one pixel of photodiode corresponds toone-color color filter.

FIG. 4 is a block diagram showing a configuration example of the signalsynthesizer means 7. In the diagram, the numeral 701 designates twohorizontal line adder means for adding image signals for two horizontalscan lines of said image signals outputted from the image memory 6(hereinafter, an image signal equivalent to the horizontal scan line iscalled simply the horizontal line or the horizontal line signal). Thenumeral 702 designates interpolation means for interpolating in thevertical direction said image signals outputted from the image memory 6.Weighting adder means 703 weights and adds the output from the twohorizontal line adder means 701 and the interpolation means 702.

FIG. 5 is a block diagram showing a configuration example of the twohorizontal line adder means 701. In the diagram, the numeral 70101designates a one-line memory which is means for delaying by onehorizontal synchronous period said image signal for one line outputtedfrom the image memory 6. The numeral 70102 designates an adder in whichthe horizontal line signal delayed at the one-line memory 70101 and thehorizontal line signal inputted into the two horizontal line adder means701 are added to cause two upper/lower lines adjacent to each other tobe added.

FIG. 6 is a block diagram showing a configuration of the interpolationmeans 702. In the diagram, the numerals 70201 and 70202 designateone-line memories which are means for delaying by one horizontalsynchronous period said image signal for one line outputted from theimage memory 6. The numerals 70203 and 70204 designate amplifier meansfor multiplying the input signal from the image memory 6 and the outputsignal of the one-line memory 70202 by a certain gain. The numeral 70205designates an adder for adding the signals multiplied by the gain at theamplifier means 70203 and 70204.

FIG. 7 is a block diagram showing a configuration of the weighting addermeans 703. In the diagram, the numeral 70301 designates synthesiscoefficient generation means for generating a coefficient k (1≧k≧0)according to the signal level for each pixel of the signals through thetwo horizontal line adder means 701 and giving k value and 1−k value tomultipliers 70302 and 70303. The multipliers 70302 and 70303 multiplythe signal through the interpolation means 702 and the signal throughthe two horizontal line adder means 701 by k value and 1−k value, theresults being added to the adder 70304 to output.

With respect to the solid-state imaging device of the present inventionconfigured as described above, the operation thereof will be explainedhereinafter.

This device is wherein it picks up two images of a short-time exposuresignal (hereinafter called the short signal) and a long-time exposuresignal (hereinafter called the long signal), and synthesizes the twoimages to pick up an image with an expanded dynamic range. The principleof such a dynamic range expansion will be explained using FIG. 8. FIGS.8( a) and 8(b) show a relationship between the brightness of a subjectwhen exposed (amount of incident light to the solid-state imagingelement) and the quantity of the signal outputted from the solid-stateimaging element. As shown in FIG. 8( a), at a long-time exposure, theamount of the charge generated on the photodiode of the solid-stateimaging element by the incident light becomes large, and thus naturallythe quantity of the outputted signal becomes large. However, the amountof the charge accumulated on the photodiode has an upper limit, and whenexceeding the upper limit, an phenomenon of saturation, that is, abroken signal occurs, so that the subject image cannot be correctlyreproduced. On the contrary, as shown in FIG. 8( b), setting theexposure time at short allows the saturation to be avoided, which inturn causes the S/N of the low brightness portion in the subject to bedegraded. Thus, using the signal obtained by a long-time exposure (thelong signal) and the signal obtained by a short-time exposure (the shortsignal), synthesizing an image consisting of the long signal for the lowbrightness portion and of short signal for the high brightness portionallows the subject to be reproduced from the low brightness portion tothe high brightness portion, thereby allowing the dynamic range of theimaging device to be expanded. At this time, after multiplying the shortsignal by a gain equivalent to the ratio in the amount of exposed lightto the long signal (the ratio in exposure time) as shown in FIG. 8( c),and synthesizing them allows the expansion of the dynamic rangeaccording to the ratio in the amount of exposed light to be achieved asshown in FIG. 8( d). For example, when the ratio in the amount ofexposure of the long signal to the short signal is 1:D, the dynamicrange can be expanded by D times.

Hereinafter, there will be explained a concrete example of the imagingdevice capable of expanding the dynamic range of the picked up imagesaccording to the above-mentioned principle.

First, using FIG. 9, a method of picking up the short signal and thelong signal will be explained. FIG. 9 is a timing chart with respect tothe exposure of a subject image, and to the read of an exposed signal inthe solid-state imaging element 3. In the chart, the item (a) indicatesa synchronous signal in the vertical direction; the item (b), a readcontrol pulse for controlling the read of the signal charge from thephotodiode of the solid-state imaging element 3; the item (c), theopening/closing state of the mechanical shutter 2; the item (d), anexposure signal on the photodiode of the solid-state imaging element 3;and the item (e), a signal outputted from the solid-state imagingelement 3.

At the short signal exposure, with the mechanical shutter 2 beingopened, using an electrical shutter function, the exposure is performedfor a required exposure time, for example, one thousandth of a second.After the exposure for one thousandth of a second is finished, thecharge accumulated on the photodiode is moved to the vertical transferCCD by the read control pulse. At this time, the solid-state imagingelement 3 should drive in the field read mode, and as explained in FIG.2( a), the charges accumulated on the photodiode are mixed on verticaltransfer CCD and read to the outside. At this time, said image signal tobe read should be only the signal for the first field. FIG. 10 shows theshort signal read in the field read mode. The number of the photodiodesin the vertical direction of the solid-state imaging element 3 is takenas N (for the sake of explanation, N is taken as even number, but notlimited to it). As shown in FIG. 10, the read short signal becomesfour-kind signals Ye+Mg, Cy+G, Ye+G and Cy+Mg obtained by adding fourcolors Ye, Cy, G and Mg to each other, respectively. The number of thelines in the vertical direction is a half of the number N in thevertical direction of photodiodes.

Then, while reading the short signal, long signal is exposed. Theexposure time of the long signal is taken as, for example, one hundredthof a second. The exposure time of the long signal should be controlledby the opening/closing of the mechanical shutter 2, and after onehundredth of a second following the start of the exposure of the longsignal, the mechanical shutter 2 is closed to complete the exposure. Inthis manner, the closing of the mechanical shutter 2 causes the exposedsignal for a long time not to be exposed extra during reading.

When the exposure of the long signal is completed, the chargeaccumulated on the photodiode is transferred to the vertical transferCCD by the read control pulse. At this time, the solid-state imagingelement 3 should drive in the frame read mode, and as explained in FIG.2( c), the charge of the photodiode equivalent to the odd number line inthe vertical direction is read by the first field. After the signal readof the first field is finished, then the charge of the photodiodeequivalent to the even number line in the vertical direction is read(the second field), whereby the long signal equivalent to one frame isread from the solid-state imaging element 3. The cycle of the verticalsynchronous signal shown in FIG. 9( a) is taken as, for example, onehundredth of a second, and the signal read for one field from thesolid-state imaging element 3 should be completed within one cycle ofthe vertical synchronous signal. FIG. 11 shows the long signal read inthe frame read mode. As shown in FIG. 11, the read long signal becomes asignal with two colors of Ye and Cy for the first field, and a signalwith two colors of G and Mg for the second field. The number of thelines in the vertical direction is a half of the number N of thephotodiodes in the vertical direction for each field, and thus combiningthe two fields causes the N-line signal equivalent to one frame to beobtained.

The performing the exposure and the signal read as described aboveallows two signals having different exposure times, that is, the shortsignal as one field image and the long signal as one frame image to beobtained. The short signal is a half in the number of horizontal linesof the long signal, so that the short signal has a small number ofpixels than the long signal.

Then, the two signals with different exposure times obtained by thesolid-state imaging element 3 are sent through the analog signalprocessing means 4, converted by the A/D converter means 5 to digitalsignals and stored for a time on the image memory 6.

The long signal and the short signal are read from the image memory 6.In reading the long signal from the image memory 6, the long signalshould be sequentially read from the leading line when assumed as oneframe image in a manner that the first line of the first field, thefirst line of the second field, followed by the second line of the firstfield are read. The long signal read from the image memory 6 is sent tothe two horizontal line adder means 701. In the two horizontal lineadder means 701, the long signals of the two upper/lower lines adjacentto each other when assumed as frame signals are added and mixed. This isbecause in synthesizing the long signal and the short signal, twosignals when being of different signal types cannot be synthesized, sothat the long signal is subject to the same processing as the pixelmixing on the vertical transfer CCD of the solid-state imaging element 3by the two horizontal line adder means 701, while for the short signal,one field image is converted by the interpolation means 702 to one frameimage.

FIG. 12( a) shows the long signal after the two upper/lower line signalsadjacent to each other are added and mixed in the two horizontal lineadder means 701; FIG. 12( b), the short signal before the interpolationprocessing; and FIG. 12( c), the short signal after the interpolationprocessing. As shown in FIGS. 12( a) and 12(c), the horizontal lineaddition processing for the long signal and the interpolation processingfor the short signal causes the long signal and the short signal to bematched in the signal type with each other.

In the interpolation means 702, the field image shown in FIG. 12( b) isconverted by the interpolation processing to the frame image shown inFIG. 12( c), which method will be explained hereinafter.

For example, when the horizontal line signal between the second line andthe third line in FIG. 12( b) is determined, it is necessary to producea horizontal line signal consisting of the signals Ye+G and Cy+Mg. Atthis time, the line consisting of the near most signals Ye+G and Cy+Mgare the second line and the fourth line, so that from both the lines, aline between the second line and the third line is determined by theinterpolation processing. However, the spatial distances between theposition at which the horizontal line signal is determined by theinterpolation processing and the second line and the fourth line are notequidistant, so that the weighting becomes necessary according to thedistance. Thus, the interpolation means 702 has a configuration suchthat of the three-line horizontal line signals inputted continuously,the upper/lower end lines except the center line are inputted into themultipliers 70203 and 70204, so that it is sufficient that the numbersto be multiplied in the multipliers 70203 and 70204 are ¼ and ¾,respectively, to be used for weighting, and the multiplied results areadded in the adder 70205.

The numbers to be multiplied in the multipliers 70203 and 70204 aredecided by the fact that the ratio in spatial distance between theposition at which the horizontal line signal is determined by theinterpolation processing and the second line and the fourth line is 1:3.

Similarly, when determining a horizontal line signal between the thirdline and the fourth line by the interpolation processing, it isnecessary to produce the horizontal line signal consisting of Ye+Mgsignal and Cy+G signal, and at this time, the lines consisting of Ye+Mgsignal and Cy+G signal are the third line and the fifth line, so thatweighting is performed according to the ratio in distance between boththe lines, thereby allowing the line between the third line and thefourth line to be determined by the interpolation processing.

With the above-mentioned processing, a signal equivalent to one frameobtained through the interpolation processing from the long signal forone frame and the short signal for one field is produced.

Means for synthesizing the long signal and the short signal tosynthesize a signal with an expanded dynamic range is the weightingadder means 703. In the weighting adder means 703, the synthesiscoefficient k according to the signal level for each pixel of the longsignal by the synthesis coefficient generation means 70301 shown in FIG.7 is determined, and the long signal and the short signal which becomesone frame image by the interpolation processing and exists on the samespatial position on the screen are synthesized in one pixel unitsaccording to the synthesis coefficient k.

FIG. 13 is an example of a method of determining the synthesiscoefficient k for each pixel from the signal level of the long signal inthe synthesis coefficient generation means 70301. As shown in FIG. 13,two thresholds Th_min and Th_max are set for the long signal; when thelong signal level is of the expression (1), that is, when the longsignal level is Th_min or less and has no possibility of saturation, thesynthesis coefficient k is taken as 0, while when the long signal levelis of the expression (2), that is, when the long signal level is Th_maxor more and the output of the solid-state imaging element is close tothe saturation level, the synthesis coefficient k is taken as 1. Thethresholds Th_min and Th_max are appropriately determined according tothe saturation characteristics of the solid-state imaging element to beused and to the S/N.

0≦long signal level≦Th_min  (1)

Th_max≦long signal level  (2)

When the long signal level is of the expression (3), that is, when thelong signal level is intermediate, as shown in FIG. 13, the synthesiscoefficient k is decided by the linear equation of the equation (4).

$\begin{matrix}{{Th\_ min} < {{long}\mspace{14mu} {signal}\mspace{14mu} {level}}\mspace{14mu} < {Th\_ max}} & (3) \\{K = {{\{ {1/( {{Th\_ max} - {Th\_ min}} )} \} \times ( {{long}\mspace{14mu} {signal}\mspace{14mu} {level}} )} - \{ {{Th\_ min}/( {{Th\_ max} - {Th\_ min}} )} \}}} & (4)\end{matrix}$

Using the synthesis coefficient k thus determined, the long signal andthe short signal are synthesized by the equation (5) for each pixel. Asignal obtained by synthesizing the long signal and the short signal istaken as a synthesis signal.

Synthesis signal=(1−k)×long signal+k×short signal×D  (5)

For example, when determining a synthesis signal (Ye+Mg) M11 from a longsignal (Ye+Mg)L11 shown in FIG. 14 and from a short signal (Ye+Mg)S11having the same spatial position as the (Ye+Mg)L11, expressing thesynthesis coefficient decided from the long signal as kill,synthesization is performed by the equation (6).

(Ye+Mg)M11=(1−k11)×(Ye+Mg)L11+k11×(Ye+Mg)S11×D  (6)

Other pixels of the synthesis signal, as with equation (6), aredetermined by the long signal and the short signal both of which existon the same spatial position.

The constant D by which the short signal is multiplied in the equations(5) and (6) is the ratio in the amount of exposed light of the longsignal to the short signal (the ratio in the exposure time), and forexample, expressing the amount of exposed light of the long signal(exposure time) as TL, and the amount of exposed light of the shotsignal (exposure time) as TS, the constant D is determined by theequation (7).

D=TL/TS  (7)

Using the long signal and the short signal in this manner, synthesizinga synthesis signal consisting of the long signal for the portion inwhich the signal level of the long signal is threshold Th#min or less,of the short signal for the portion in which signal level thereof isthreshold Th#max or more, that is, the output of the solid-state imagingelement 3 is near saturation (the portion in which the brightness of thepicked up image is high, so that the signal would be broken if in normalcondition), and of a signal for the intermediate brightness portion inwhich the long signal and the short signal are weighted and added allowsthe dynamic range of the picked up image signal to be expanded.

However, of the synthesis signals whose dynamic range is expanded, theportion consisting of the long signal is originally one frame imagesignal, so that the image resolution is high. On the other hand, theportion consisting of the short signal is synthesized from one fieldimage signal, so that the image resolution is lower than the portionconsisting of the long signal. However, generally, it is unusual todevelop a condition in which the signal level of the whole image becomesclose to saturation, and even in such a condition, the amount ofincident light is limited by stopping down the optical aperture, so thatthe signal level of the whole image becomes hardly a level close tosaturation, and thus a condition in which the most of the picked upimage is occupied by the portion consisting of the short signal canhardly occur practically. When an image is expressed by limitedgradations, a high brightness portion, that is, a high signal levelportion is often allocated with fewer gradations than a low/intermediatebrightness portions. Hence, the degradation in the resolution of theportion consisting of the short signal is not so prominent, so that itis considered that synthesizing the long signal and the short signaleven by the above-mentioned method causes a synthesis image with aresolution equivalent to the number of pixels of CCD to be obtained.

As described above, a signal synthesized by the signal synthesizer means7 is subject to processings such as separating of brightness signal fromcolor signal, noise removing, edge enhancing, gamma correcting, matrixoperating, and encoding to a specific format in the digital signalprocessing means 8. The signal processing in the digital signalprocessing means 8 is not directly related to the object of the presentinvention, so that a detailed explanation will be omitted.

As described above, the above-mentioned solid-state imaging devicecontrols the exposure of the solid-state imaging element 3 and thesignal read mode, picks up the short-time exposure signal for one fieldand the long-time exposure signal for one frame, and synthesizes thesesignals, thereby allowing an image whose dynamic range is expanded whilehaving a resolution equivalent to the number of pixels of CCD to bepicked up. Further, the solid-state imaging element used with thepresent solid-state imaging device can use IT-CCD generally used inconsumer solid-state imaging devices, so that the device can beconfigured at a low cost without using a plurality of solid-stateimaging elements or a special solid-state imaging element.

In addition to a procedure in that the exposure of the short signal istaken as the first exposure and the exposure of the long signal is takenas the second exposure, and with these exposures, the read sequence ofthe charge accumulated on the photodiode is set as described above,there may be a procedure in that the exposure of the long signal istaken as the first exposure and the exposure of the short signal istaken as the second exposure, and the read sequence is made reverse.

ANOTHER MODE FOR CARRYING OUT THE INVENTION

In contrast to Embodiment 1, other embodiments 2 through 8 will beexplained hereinafter.

Embodiment 2

The solid-state imaging device in Embodiment 2 of the present inventionis different from Embodiment 1 of the present invention shown in FIG. 1in the configuration of the weighting adder means (in Embodiment 2,distinguished from it by numbering 704) and in the processing performedin the means. Hereinafter, the explanation of the processing contentssimilar to Embodiment 1 of the present invention will be omitted, andonly the portion different from Embodiment 1 of the present inventionwill be explained.

FIG. 15 is a block diagram of weighting adder means 704 in Embodiment 2of the present invention. In the diagram, the numeral 70401 designatesbrightness signal extraction means for extracting a brightness signalcomponent from the long signal passing through the two horizontal lineadder means 701. The numeral 70402 designates synthesis coefficientgeneration means for generating a coefficient k (1≧k≧0) according to thebrightness signal level of the brightness component of the long signalthrough the brightness signal extraction means 70401 and giving k valueand 1−k value to multipliers 70403 and 70404. The multipliers 70403 and70404 multiply the short signal through the interpolation means 702 andthe long signal through the two horizontal line adder means 701 by kvalue and 1−k value, the results being added to the adder 70405 tooutput.

FIG. 16 is a block diagram showing a configuration example of thebrightness signal extraction means 70401. In the diagram, the numeral704011 designates means for delaying by one pixel period the inputsignal. The numeral 704012 designates an adder for adding said imagesignal delayed in the one pixel delay means 704011 and said image signalinputted into the brightness signal extraction means 70401 to add twopixels adjacent in the horizontal direction to each other, therebyextracting only the low-pass component of the signal. The low-passcomponent of the signal extracted by the brightness signal extractionmeans 70401 is equivalent to the brightness signal of said image signal.

With respect to the solid-state imaging device of Embodiment 2 of thepresent invention configured as described above, the operation thereofwill be explained hereinafter.

Unlike Embodiment 1 of the present invention, in Embodiment 2 of thepresent invention, the synthesis coefficient used in synthesizing thelong signal and the short signal is decided according to the signallevel of the brightness signal extracted from the long signal.

Hence, the brightness signal extraction means 70401 as means forextracting the brightness signal from the long signal is provided in theweighting adder means 704.

In the brightness signal extraction means 70401, of the outputs of thetwo horizontal line adder means 701, two pixel signals adjacent in thehorizontal direction to each other are sequentially added, whereby thebrightness component of the long signal (hereinafter called the longbrightness signal) is extracted on the basis of the following equation(8).

Brightness component(brightness signal)=Ye+Mg+Cy+G  (8)

For example, when determining the long brightness signal YL11 from thelong signal (Ye+Mg) L11 and the long signal (Cy+G) L12, (Ye+Mg) L11 and(Cy+G) L12 are added. Similarly, when determining the long brightnesssignal YL12, (Cy+G) L12 and (Ye+Mg) L13 are added.

A method of deciding a synthesis coefficient on the basis of thebrightness signal extracted from the long signal (long brightnesssignal) will be explained hereinafter.

FIG. 18 is an example of a method of determining the synthesiscoefficient k for each pixel from the signal level of the longbrightness signal in the synthesis coefficient generation means 70402.As shown in FIG. 18, two thresholds Th_min′ and Th_max′ are set for thelong brightness signal level; when the long brightness signal level isof the expression (9), that is, when the brightness level of a subjectis as low as Th_min′ or less, the synthesis coefficient k is taken as 0,while when the long brightness signal level is of the expression (10),that is, when the brightness level of a subject is as high as Th_max′ ormore, the synthesis coefficient k is taken as 1. The thresholds Th_min′and Th_max′ are appropriately determined according to the saturationcharacteristics of the solid-state imaging element to be used and to theS/N.

0≦long brightness signal level≦Th_min′  (9)

Th_max′≦long brightness signal level  (10)

When the long brightness signal level is of the expression (11), thatis, when the brightness is intermediate between low brightness and highbrightness, as shown in FIG. 18, the synthesis coefficient k is decidedby the linear equation of the equation (12).

$\begin{matrix}{{{{Th\_ min}’} < {{long}\mspace{14mu} {brightness}\mspace{14mu} {signal}\mspace{14mu} {level}} < {Th\_ max}}’} & (11) \\  {{{  {K = \{ {{1/( {Th\_ max}’ } - {Th\_ min}}’ } ) \} \times ( {{long}\mspace{14mu} {brightness}\mspace{14mu} {signal}\mspace{14mu} {level}} )} - {\{ {Th\_ min}’ /( {Th\_ max}’ } - {Th\_ min}}’} ) \} & (12)\end{matrix}$

Using the synthesis coefficient k thus determined, the long signal andthe short signal are synthesized by the equation (5) for each pixel. Asignal obtained by synthesizing the long signal and the short signal istaken as a synthesis signal.

For example, when determining a synthesis signal (Ye+Mg) M11 from a longsignal (Ye+Mg)L11 shown in FIG. 19 and from a short signal (Ye+Mg)S11having the same spatial position as the (Ye+Mg)L11, on the basis of asynthesis coefficient (taken as ky11) decided from the long brightnesssignal YL11 having the same spatial position as these two signals,synthesization is performed by the equation (13).

(Ye+Mg)M11=(1−ky11)×(Ye+Mg)L11+ky11×(Ye+Mg)S11×D  (13)

Other pixels of the synthesis signal, as with equation (13), aredetermined from the long signal and the short signal both of which existon the same spatial position.

The constant D by which the short signal is multiplied in the equation(13) is the ratio in the amount of exposed light of the long signal tothe short signal (the ratio in the exposure time), as with Embodiment 1of the present invention, is determined by the equation (7).

Using the long signal and the short signal in this manner, synthesizinga synthesis signal consisting of the long signal for the low brightnessportion, of the short signal for the high brightness portion, and of asignal for the intermediate brightness portion between the lowbrightness portion and the high brightness portion in which the longsignal and the short signal are weighted and added allows the dynamicrange of the picked up image signal to be expanded.

The brightness signal can be said as a low frequency component extractedfrom the long signal, so that when determining a synthesis coefficienton the basis of the brightness signal, an effect of the noise componentin the long signal on the synthesis coefficient decision can be reduced.

As described above, the solid-state imaging device of Embodiment 2 ofthe present invention also controls the exposure of the solid-stateimaging element 3 and the signal read mode, picks up the short-timeexposure signal for one field and the long-time exposure signal for oneframe, and synthesizes these signals, thereby allowing an image whosedynamic range is expanded while having a resolution equivalent to thenumber of pixels of CCD to be picked up. Further, the solid-stateimaging element used with the present solid-state imaging device can useIT-CCD generally used in consumer solid-state imaging devices, so thatthe device can be configured at a low cost without using a plurality ofsolid-state imaging elements or a special solid-state imaging element.

Embodiment 3

FIG. 20 is a block diagram of the solid-state imaging device inEmbodiment 3 of the present invention. In the diagram, thefunction/operation of the optical lens 1, the mechanical shutter 2 alsoused for the optical aperture, the solid-state imaging element 3, theanalog signal processing means 4, the A/D converter means 5, the imagememory 6, the shutter drive means 9, the solid-state imaging elementdrive means 10, the two horizontal line adder means 701, the brightnesssignal extraction means 70401, and the interpolation means 702 issimilar to Embodiments 1 and 2 of the present invention, so that thesame numerals as in FIGS. 1 through 19 are assigned to them, and thusthe explanation will be omitted.

Explaining the components other than described above in the blockdiagram shown in FIG. 20, the numeral 12 designates a brightness signalinterpolation means for applying the interpolation processing to theoutput of the brightness signal extraction means 70401, and thebrightness signal synthesizer means 13 synthesizes the outputs ofbrightness signal extraction means 70401 and the brightness signalinterpolation means 12. The brightness signal inputted into thebrightness signal interpolation means 12 is extracted from the shortsignal, so that the brightness signal is called the short brightnesssignal, while the brightness signal extracted from the long signal iscalled the long brightness signal. Thus, the signal inputted from thebrightness signal extraction means 70401 into directly the brightnesssignal synthesizer means 13 becomes the long brightness signal, whilethe signal inputted from the brightness signal interpolation means 12into the brightness signal synthesizer means 13 becomes the signal ofthe short brightness signal after being interpolated.

A signal synthesizer means 14 synthesizes the outputs of the two lineadder means 701 and the interpolation means 702. One-line memories 15,16, 17 and 18 are delay means for one horizontal synchronous periodrequired in synchronizing the output of the signal synthesizer means 14,and from the horizontal line signal having the total five linesconsisting of the four line output of the one-line memories 15, 16, 17and 18 and of one-line output of the signal synthesizer means 14, asignal having a red (R) component and a blue (B) component both on thesame spatial position is obtained by synchronization means 19.

A signal having the brightness signal obtained by the brightness signalsynthesizer means 13, and the signal with the red (R) component and thesignal with the blue (B) component obtained by the synchronization means19 is subject to processings such as noise removing, edge enhancing,matrix operating, and encoding to a specific format in a digital signalprocessing means 20. The operation mode and operation timing of all ofthe above-mentioned components including these means should becontrolled integrally by a system control means 21.

FIG. 21 is a block diagram showing a configuration of the brightnesssignal interpolation means 12. In the diagram, the numeral 1201 is aone-line memory which is means of delaying by one horizontal synchronousperiod the one line portion of said image signal outputted from thebrightness signal extraction means 70401. The numerals 1201 and 1203designate amplifier means for multiplying the input through the one-linememory 1201 and the signal inputted through the brightness signalextraction means 70401 into the brightness signal interpolation means 12by a certain gain. The numeral 1204 designates an adder for adding thesignals multiplied by the gain at the amplifier means 1202 and 1203.

FIG. 22 is a block diagram showing a configuration of the brightnesssignal synthesizer means 13. In the diagram, the numeral 1301 designatessynthesis coefficient generation means for generating a coefficient k(1≧k≧0) according to the signal level for each pixel of the longbrightness signals through the brightness signal extraction means 70401and giving k value and 1−k value to multipliers 1302 and 1303. Themultipliers 1302 and 1303 multiply the short brightness signal and thelong brightness signal through the two horizontal line adder means 701by k value and 1−k value, the results being added to the adder 70304 tooutput.

FIG. 23 is a block diagram showing a configuration of the signalsynthesizer means 14. In the diagram, the numerals 1401 and 1402designate multipliers for multiplying the short signal and the longsignal after the two horizontal line addition by coefficients k and 1−ksupplied from the brightness signal synthesizer means 13. The resultsare added to the adder 1403 to output.

FIG. 24 is a block diagram showing a configuration of the synchronizermeans 19. In the diagram, the numeral 1901 designates a selector forselecting three signals among inputted signals and outputting them tooutputs A, B and C; and the numerals 1902 and 1903 designate amplifiersfor multiplying the signals outputted from the outputs B and C by aconstant, the signals after amplified being added by an adder 1904. Thenumeral 1905 designates a selector for dividing the output A of theselector 1901 and the output of the adder 1904 among the outputs D andE. The selection of the output destination of the signals by theselectors 1901 and 1905 is divided by the color component of the signalas described later.

With respect to the solid-state imaging device of Embodiment 3 of thepresent invention configured as described above, the operation thereofwill be explained hereinafter.

The third embodiment of the present invention is the same as Embodiments1 and 2 of the present invention in that two images of the short-timeexposure signal (the short signal) and the long-time exposure signal(the long signal) are picked up to be synthesized, thereby picking up animage with an expanded dynamic range. However, Embodiment 3 of thepresent invention is wherein individually for the brightness signal andfor the signal later processed as a color signal, the short-timeexposure signal (the short signal) and the long-time exposure signal(the long signal) are synthesized.

Hence, in Embodiment 3 of the present invention, as with Embodiment 1 ofthe present invention, for the long signal read from the image memory 6when assumed as frame signal, in the two horizontal line adder means701, the long signals of the two upper/lower lines adjacent to eachother are added and mixed. This is a step to match the long signal withthe fact that the short signals are image mixed on the vertical transferCCD of the solid-state imaging device of Embodiment 3.

In the brightness signal extraction means 70401, as with Embodiment 2 ofthe present invention, of the outputs of the two horizontal line addermeans 701, two pixel signals adjacent in the horizontal direction toeach other are sequentially added, whereby the brightness component ofthe long signal (hereinafter, called the long brightness signal) isextracted on the basis of the equation (8).

For example, when determining the long brightness signal YL11 from thelong signal (Ye+Mg) L11 and the long signal (Cy+G) L12, (Ye+Mg) L11 and(Cy+G) L12 are added. Similarly, when determining the long brightnesssignal (Cy+G) YL12, L12 and (Ye+Mg) L13 are added.

Then, for the short signal read from the image memory 6, first thebrightness signal is determined at the brightness signal extractionmeans 70401, as with the long signal.

FIG. 25 shows the long brightness signal; and FIG. 26 shows the shortbrightness signal.

As shown in FIG. 26, the short signal was one-field signal, so that theshort brightness signal also naturally is one-field brightness signal.Thus, a means for converting the one-field short brightness signal toone-frame signal to make the short brightness signal identical in thesignal type to the long brightness signal is the brightness signalinterpolation means 12.

More specifically, the brightness signal interpolation means 12determines an addition mean value of the two continuous lines by settingthe gain multiplied in the amplifier means 1202 and 1203 at 0.5 andtakes it an interpolation signal. FIG. 27 shows the short brightnesssignal after the interpolation processing.

With the above-mentioned processing, the brightness signal obtained fromone-frame long signal (the long brightness signal) and the brightnesssignal equivalent to one frame obtained through the interpolationprocessing from one-field short signal (the short brightness signal) areobtained. The reason that one-frame short brightness signal issynthesized from one-field short brightness signal in this manner isthat in synthesizing the short signal and the long signal to expand thedynamic range, when the short signal is still one-field signal, thehorizontal line is short, so that the short signal cannot be synthesizedwith the long signal as the one-frame signal.

A means for synthesizing the long brightness signal and the shortbrightness signal to synthesize the brightness signal with an expandeddynamic rage is the brightness signal synthesizer means 13. Inbrightness signal synthesizer means 13, the synthesis coefficient kaccording to the signal level for each pixel of the long signal by thesynthesis coefficient generation means 1301 shown in FIG. 22 isdetermined, and the long brightness signal and the short brightnesssignal which exists on the same spatial position on the screen aresynthesized in one pixel units according to the synthesis coefficient k.

As one example of a method of determining the synthesis coefficient kfor each pixel from the signal level of the long brightness signal, thesame method as Embodiment 2 of the present invention is considered, sothat the explanation thereof will be omitted.

Using the synthesis coefficient k thus determined, the long brightnesssignal and the short brightness signal are synthesized for each pixel bythe equation (14). A signal obtained by synthesizing the long brightnesssignal and the short brightness signal is taken as the synthesisbrightness signal.

$\begin{matrix}{{{Synthesis}\mspace{14mu} {brightness}\mspace{14mu} {signal}} = {{( {1 - k} ) \times {long}\mspace{14mu} {brightness}\mspace{14mu} {signal}} + {k \times {short}\mspace{14mu} {brightness}\mspace{14mu} {signal} \times D}}} & (14)\end{matrix}$

For example, when determining a synthesis brightness signal YM11 from along brightness signal YL11 shown in FIG. 28 and from a short brightnesssignal YS11 having the same spatial position as the YL11, a synthesiscoefficient decided from the long brightness signal YL11 is taken askill, whereby synthesization is performed by the equation (15).

YM11=(1−k11)×YL11+k11YS11×D  (15)

Other pixels of the synthesis brightness signal, as with the equation(15), are determined from the long brightness signal and the shortbrightness signal both of which exist on the same spatial position.

The constant D by which the short brightness signal is multiplied in theequations (14) and (15) is the ratio in the amount of exposed light ofthe long signal to the short signal (the ratio in the exposure time), aswith Embodiment 1 of the present invention, and is determined by theequation (7).

Using the long brightness signal and the short brightness signal in thismanner, synthesizing a synthesis brightness signal consisting of thelong signal for the low brightness portion, of the short brightnesssignal for the high brightness portion, and of a signal for theintermediate brightness portion between the low brightness portion andthe high brightness portion in which the long brightness signal and theshort brightness signal are weighted and added allows the dynamic rangeof the picked up image brightness signal to be expanded.

However, of the synthesis signals whose dynamic range is expanded, theportion consisting of the long brightness signal is originally one frameimage signal, so that the image resolution is high. On the other hand,the portion consisting of the short brightness signal is synthesizedfrom one field image signal, so that the image resolution is lower thanthe portion consisting of the long brightness signal. However,generally, it is unusual to develop a condition in which the whole imagebecomes high in brightness, and even in such a condition, the amount ofincident light is limited by stopping down the optical aperture, so thatthe signal level of the whole image becomes hardly high in brightness,and thus a condition in which the most of the picked up image isoccupied by the portion consisting of the short brightness signal canhardly occur practically. When an image is expressed by limitedgradations, a high brightness portion is allocated with fewer gradationsthan a low/intermediate brightness portions. Hence, the degradation inthe resolution of the portion consisting of the short brightness signalis not so prominent, so that it is considered that synthesizing the longbrightness signal and the short brightness signal even by theabove-mentioned method causes a synthesis image with a resolutionequivalent to the number of pixels of CCD to be obtained.

The above-mentioned description is for the contents of the processingrelated to the dynamic range expansion by the brightness signalsynthesization. The processing related to the color signal will beexplained hereinafter.

The short signal read from the image memory 6, and the long signal towhich the upper/lower lines adjacent to each other are added in the twohorizontal line adder means 70 are subject to a synthesis processing inthe signal synthesizer means 14 to expand the dynamic range of the colorsignal.

The short signal, which is one field signal, is different in the signaltype from the long signal. Thus, as with Embodiment 1 of the presentinvention, the one field image is converted by the interpolation means702 to the one frame image.

The long signal after being subject to the addition and mixing of theupper/lower lines adjacent to each other in the two horizontal lineadder means 701, and the short signal having been subject to theinterpolation processing in the interpolation means 702 are as shown inFIGS. 12( a) and 12(b), and as with Embodiment 1 of the presentinvention, the two horizontal line addition processing for the longsignal and the interpolation processing for the short signal causes thelong signal and the short signal to be matched in the signal type witheach other.

The synthesis of the long signal and short signal inputted into thesignal synthesizer means 14, as with Embodiment 2 of the presentinvention, is performed for each pixel by the synthesis coefficient kused when the long brightness signal and the short brightness signal aresynthesized both of which are matched spatially in the position with thelong signal and short signal inputted into the signal synthesizer means14, and by D determined by the equation (7). A signal synthesized in thesignal synthesizer means 14 is called the synthesis signal.

The above-mentioned description is for the synthesis processing toexpand the dynamic range of the color signal.

The synthesis signal determined by the signal synthesizer means 14 has aconfiguration in which a line on which the pixels of Ye+Mg and Cy+G arearranged in the horizontal direction and a line on which Ye+G and Cy+Mgare arranged in the horizontal direction are repeated in verticaldirection at two line cycles, so that expressing red, green and blue ofthree primary colors as R, G and B, respectively, there are obtained acolor signal as 2R−G having R component by the equation (16) from theline on which the pixels of Ye+Mg and Cy+G are arranged, and a colorsignal as 2B−G having B component by the equation (17) from the line onwhich Ye+G and Cy+Mg are arranged.

(Ye+Mg)−(Cy+G)≈2R−G  (16)

(Cy+Mg)−(Ye+G)≈2B−G  (17)

This is the so-called color difference line sequence, so that there isobtained only the color signal of one of either 2R−G having R componentor 2B−G having B component for one horizontal line signal. Thus, toobtain a signal having both R component and B component for onehorizontal line signal, a synchronization processing is performed by theline memories 15, 16, 17, 18 and the synchronization means 19.

Hereinafter, there will be explained concrete contents of thesynchronization processing by the line memories 15, 16, 17, 18 and thesynchronization means 19. Inputted into the synchronization means 19 isthe horizontal line signal of five lines continuing from the signalsynthesizer means 14 and the line memories 15, 16, 17, 18. With a signalsynthesized in the signal synthesizer means 14 taken as the synthesissignal in FIG. 29( a), suppose that the signal of five lines inputtedinto the synchronization means 19 is a signal of the third line throughthe seventh line shown in FIG. 29( b). At this time, assuming that asubject of the synchronization processing is a horizontal line signallocated on the center of inputted five lines, when the horizontal linesignal of the fifth line of FIG. 29( b) is attempted to be subject tothe synchronization processing, the fifth line is a signal correspondingto 2R−G having R component, so that it is sufficient to produce 2B−Ghaving B component by the interpolation processing from peripheralhorizontal line signals. Thus, in the synchronization means 19 shown inFIG. 24, the selector 1901 outputs the signal of the fifth line to theoutput A, and the signals corresponding to 2B−G of the third line andthe seventh line to the output B and the output C. Taking a gainmultiplied in the adder means 1902, 1903 as 0.5, adding the multipliedresults in the adder 1904 causes mean addition results of the third lineand the seventh line to be obtained. The mean addition results and thesignal of the fifth line as the output of the output A of the selector1901 are inputted into the selector 1905, where an output destination isselected, and the horizontal line signal of the fifth line correspondingto 2R−G is outputted to the output D, while the mean addition results ofthe third line and the seventh line corresponding to 2R−G are outputtedto the output E. Such an operation allows the signal corresponding to2R−G having R component, and the signal corresponding to 2B−G having Bcomponent to be obtained at a spatial position at which the fifth lineexists. Similarly, for example, when with the signal of the fifth linethrough the ninth line inputted into the synchronization means 19, thehorizontal line signal of the seventh line is attempted to be subject tothe synchronization processing, the seventh line is a signalcorresponding to 2B−G having B component, so that it is sufficient toproduce 2R−G having R component by the interpolation processing fromperipheral horizontal line signals. Thus, in the synchronization means19 shown in FIG. 24, the selector 1901 outputs the signal of the seventhline to the output A, and the signals corresponding to 2R−G of the fifthline and the ninth line to the output B and the output C. Taking a gainmultiplied in the adder means 1902, 1903 as 0.5, adding the multipliedresults in the adder 1904 causes mean addition results of the fifth lineand the ninth line to be determined. The mean addition results and thesignal of the seventh line as the output of the output A of the selector1901 are inputted into the selector 1905, where an output destination isselected, and the horizontal line signal of the seventh linecorresponding to 2B−G is outputted to the output E, while the meanaddition results of the fifth line and the ninth line corresponding to2R−G are outputted to the output D. Such an operation allows the signalcorresponding to 2R−G having R component, and the signal correspondingto 2B−G having B component to be obtained at a spatial position at whichthe seventh line exists. The selection of the input/output signal andthe like should be performed automatically or by the control of thesystem control means 21, so that the above-mentioned processing isperformed according to the input signal in the synchronization means 19.

As described above, a synthesis brightness signal synthesized by thesignal synthesizer means 13, and the signal corresponding to 2R−G havingR component and the signal corresponding to 2B−G having B componentobtained in the synchronization means 19 are subject to processings suchas noise removing, edge enhancing, gamma correcting, matrix operating,and encoding to a specific format in the digital signal processing means20. The signal processing in the digital signal processing means 20 isnot directly related to the object of the present invention, so that adetailed explanation will be omitted.

As described above, the solid-state imaging device of Embodiment 3 ofthe present invention controls the exposure of the solid-state imagingelement 3 and the signal read mode, picks up the short-time exposuresignal for one field and the long-time exposure signal for one frame,and synthesizes these signals, thereby allowing an image whose dynamicrange is expanded while having a resolution equivalent to the number ofpixels of CCD to be picked up. Further, the solid-state imaging elementused with the present solid-state imaging device can use IT-CCDgenerally used in consumer solid-state imaging devices, so that thedevice can be configured at a low cost without using a plurality ofsolid-state imaging elements or a special solid-state imaging element.

(3) Embodiment 4

Major different points of the solid-state imaging device in Embodiment 4of the present invention from Embodiment 3 of the present invention asshown in FIG. 20 exist in that, added to the solid-state imaging devicein Embodiment 4 of the present invention is means 22 of thinning out theoutput of the two horizontal line adder means 70401, in connection withwhich the interpolation means 702 and one-line memories 17, 18 areeliminated, and the configuration/function of the signal synthesizermeans, the synchronizer means, the digital signal processing means, andthe system control means are different from those in Embodiment 3 (inthe embodiment of the present invention, they are numbered as the signalsynthesizer means 23, the synchronizer means 24, the digital signalprocessing means 25, and the system control means 26 to distinguish).The explanation of the processing contents as with Embodiment 3 of thepresent invention will be omitted, and only the parts different fromEmbodiment 3 of the present invention will be explained hereinafter.FIG. 30 is a block diagram of the solid-state imaging device inEmbodiment 4 of the present invention. In the diagram, the thinning outmeans 22 thins the horizontal line signal out of the output of the twohorizontal line adder means 701, and converts one-frame signal toone-field signal. The signal synthesizer means 23 synthesizes theoutputs of the thinning out means 22 and the image memory 6 on the basisof the synthesis coefficient k determined at the brightness signalsynthesizer means 13. The synchronizer means 24 processes synchronouslythe output of the signal synthesizer means 23.

The brightness signal obtained by the brightness signal synthesizermeans 13, the signal having red (R) component obtained by thesynchronizer means 24, and the signal having blue (B) component aresubject to processings such as noise removing, edge enhancing, matrixoperating, and encoding to a specific format in the digital signalprocessing means 25. The operation mode and operation timing of all ofthe above-mentioned components including these means should becontrolled integrally by the system control means 26.

FIG. 31 is a block diagram showing a configuration of the synchronizermeans 24. The numerals 2401, 2402 designate amplifier means formultiplying the signals passing through the signal synthesizer means 23and the one-line memory 16 by a constant, the signals after amplifiedbeing added by an adder 2403. The numeral 2404 designates a selector fordividing the output of the one-line memory 15 and the output of theadder 2403 among the outputs D and E. The selection of the outputdestination of the signals by the selector 2404 should be divided by thecolor component of the signal as described later. With respect to thesolid-state imaging device of Embodiment 4 of the present inventionconfigured as described above, the operation thereof will be explainedhereinafter.

As described in Embodiment 3 of the present invention, the output of thetwo horizontal line adder means 701 is the long signal as one-framesignal. However, the short signal stored in the image memory 6 isone-field image, so that when the signal is still in this condition, thelong signal and the short signal cannot be synthesized in the signalsynthesizer means 23. Thus, in Embodiment 4 of the present invention,the short signal was converted by the interpolation processing toone-frame signal.

In Embodiment 4 of the present invention, utilizing the fact that evenwhen the color signal does not have the amount of information to thesame extent as the brightness signal, there is no problem in the imagequality, on the contrary to Embodiment 3 of the present invention, thelong signal as one-frame signal is subject to the thinning outprocessing in the vertical direction, whereby the long signal isconverted to one-field image, which is synthesized with the short signalin synchronizer means 24. More specifically, the even number lines ofthe long signal after two lines addition as shown in FIG. 12( a) arethinned out by the thinning out means 22, whereby the long signalinputted into the signal synthesizer means 23 is converted to theone-field image. The long signal after thinning out exhibits the sametype as the short signal as shown in FIG. 12( b).

The synthesis of the long signal as one-field signal and short signalinputted into the signal synthesizer means 23, as with Embodiment 3 ofthe present invention, is performed for each pixel by the synthesiscoefficient k used when the long brightness signal and the shortbrightness signal are synthesized both of which are matched spatially inthe position with the long signal and short signal, and by D determinedby the equation (7). A signal synthesized in the signal synthesizermeans 23 is called the synthesis signal.

Then, although the synchronization processing is performed in thesynchronizer means 24, unlike Embodiment 3 of the present invention, thesynthesis signal is one-field signal, so that the signal inputted intothe synchronizer means 24 may be, for example, of three lines from thesecond line to the fourth line as shown in FIG. 32. From the signal ofthree lines, as with Embodiment 3 of the present invention, there can beobtained the signal corresponding to 2R−G having R component, and thesignal corresponding to 2B−G having B component. For example, to obtainthe signal corresponding to 2R−G having R component, and the signalcorresponding to 2B−G having B component, it is sufficient the signalsof the second line and the fourth line are added and averaged tosynthesize a signal corresponding to 2B−G.

The two signals obtained by the synchronizer means 24 are processed inthe digital signal processing means 25 in the same manner as Embodiment3 of the present invention, while in Embodiment 4 of the presentinvention, the synthesis signal synthesized in the signal synthesizermeans 23 is one-field signal, so that it is needless to say that thesynthesis signal is converted to a frame image in the digital signalprocessing means 25 if required.

As described above, the solid-state imaging device of Embodiment 4 ofthe present invention, as with Embodiment 3 of the present invention,also controls the exposure of the solid-state imaging element 3 and thesignal read mode, picks up the short-time exposure signal for one fieldand the long-time exposure signal for one frame, and synthesizes thesesignals, thereby allowing an image whose dynamic range is expanded whilehaving a resolution equivalent to the number of pixels of thesolid-state imaging element to be picked up. Further, Embodiment 4 ofthe present invention processes the color signal as a field signal,thereby allowing the number of required one-line memories to be reducedand the device to be configured at a lower cost.

In Embodiment 1 of the present invention, the short signal is taken as aone-field image read in the field read mode, but is not limited to sucha configuration, and for example, a configuration is considered in whichthe signal is read by thinning out the horizontal line signal in thevertical direction. By way of example, a configuration is considered inwhich when the short signal is read from the solid-state imaging element3 as shown in FIG. 33, one-line signal is read for each three lines inthe vertical direction. In this case, the short signal is read withoutmixing of charges accumulated on the two upper/lower photodiodes in thesolid-state imaging element, thereby making the two horizontal lineaddition processing unnecessary. In the interpolation processing by theinterpolation means 702 shown in FIG. 4, it is necessary to perform theinterpolation processing in a manner to match the number of horizontallines of the short signal with the long signal. That is, in theinterpolation means 702, a horizontal line signal for two lines isproduced by the interpolation processing between horizontal line signalsof the short signal. This causes the short signal and the long signal tohave the same signal type, thereby allowing both the signals to besynthesized by the weighting addition means 703 shown in FIG. 4. In thiscase, the synthesis coefficient k may be determined from the signallevel of each pixel of the long signal to which the two upper/lowerlines are not added, by a method, for example, that shown in FIG. 13.When the short signal is read by thinning out the horizontal line signalin this manner, the two horizontal line addition processing for the longsignal is described to be unnecessary, but the processing is not limitedto such a configuration, and a configuration is also considered in whichthe long signal and the short signal are subject to the two horizontalline addition processing and then to the synthesis processing.

In Embodiment 1 of the present invention, the two signals having adifferent amount of exposure are taken as the short signal as one-fieldsignal and as the long signal as one-frame image, but are not limited tosuch a configuration, and may be taken as the long signal as one-fieldsignal and as the short signal as one-frame image depending on theapplication of the solid-state imaging element. In this case, it issufficient to provide a configuration in which as shown in FIG. 34, thelong signal is subject to the interpolation processing in the verticaldirection by the interpolation means 702, while the short signal issubject to the addition of the two upper/lower lines adjacent to eachother by the two horizontal line addition means 701, and in which thesynthesis coefficient used in the weighting addition means 703 isdetermined from the long signal after the interpolation processing. Aconfiguration is also considered in which the synthesis coefficient isdetermined from the long signal before the interpolation processing, inwhich case, it is sufficient that the corresponding long signal does notexist on the even-number lines of the short signal as shown in FIG. 34(a), and thus the synthesis coefficient k cannot be determined, so thatthe synthesis coefficient on the position of the even-number lines ofthe short signal is determined from the synthesis coefficient determinedfrom the horizontal line of the long signal existing on the sameposition as the upper/lower lines of the even-number lines of the shortsignal. In this manner, the synthesis coefficient is determined from thelong signal as one-field image and the short signal as one-frame image,thereby allowing an expanded dynamic range image with a high resolutionat a high-brightness part to be picked up.

In Embodiments 1 and 2 of the present invention, the interpolation means702 uses two one-line memories and performs the interpolation processingfrom the signal for two horizontal lines, but is not limited to suchconfiguration, and for example, a configuration is also considered inwhich the means uses a large number of one-line memories and performsthe interpolation processing by a high-rank interpolation processingfrom a large number of horizontal line signals. A configuration is alsoconsidered in which outputting repeatedly twice the one-horizontal linesinputted as shown in FIG. 35 causes the number of horizontal lines tobecome double, that is, the so-called previous-value interpolation to beperformed.

In Embodiment 1 of the present invention, in the signal synthesizermeans 7, the synthesis coefficient k for synthesizing the long signaland the short signal is determined for each pixel of the long signal,but the means is not limited to such a configuration, and for example, aconfiguration is also considered in which the synthesis coefficient k isdetermined for each pixel from the mean value, minimum value, maximumvalue or intermediate value of the signal level of a plurality ofpixels, or in which of the values of k determined for each pixel, themean value, minimum value, maximum value or intermediate value of kdetermined from a plurality of pixels is taken as the synthesiscoefficient for each pixel.

In Embodiment 1 of the present invention, in the signal synthesizermeans 7, a configuration is also considered in which a synthesiscoefficient for a block consisting of a plurality of pixels isdetermined to perform synthesization. For example, in FIG. 36, when thelong signal (Ye+Mg)L11 and (Cy+G)L12, and the long signal (Ye+Mg)L13 and(Cy+G)L14 are taken as one block, respectively, while the short signal(Ye+Mg)S11 and (Cy+G)S12, and the short signal (Ye+Mg)S13 and (Cy+G)S14existing on the same position as the long signals are taken as oneblock, respectively, it is possible to determine a synthesis coefficientfor each block in two pixel units and perform synthesization. At thistime, for example, the synthesis of the block consisting of the longsignal (Ye+Mg)L11 and (Cy+G)L12 and the block consisting of the shortsignal (Ye+Mg)S11 and (Cy+G)S12 is performed, when the synthesiscoefficient of these blocks is taken as kb11, as in the equation (18),respectively. ((Ye+Mg)M11, (Cy+G)M12 are signals after the synthesis)

(Ye+Mg)M11=(1−kb11)×(Ye+Mg)L11+kb11×(Ye+Mg)S11×D

(Cy+G)M12=(1−kb11)×(Cy+G)L12+kb11×(Cy+G)S12×D  (18)

In this case, it is sufficient that k determined by the method shown inFIG. 13 from either the signal level of the long signal (for example,(Ye+Mg)L11 and (Cy+G)L12) included in the block, or at least one of themean value, minimum value, maximum value and intermediate value of thelong signal (for example, (Ye+Mg)L11 and (Cy+G)L12) included in theblock is taken as the synthesis coefficient kb11 of the block. Aconfiguration is also considered in which one of the mean value, minimumvalue, maximum value and intermediate value of the k values (forexample, k1, k2 in FIG. 36) for each pixel determined by the methodshown in FIG. 13 from each signal level of the long signal included inthe block is taken as the synthesis coefficient kb11 of the block. It isneedless to say that the number of pixels in the block is not limited totwo pixels.

In Embodiment 1 of the present invention, in the signal synthesizermeans 7, a configuration is also considered in which a block consistingof a plurality of pixels is provided, and a synthesis coefficientdetermined by the method shown in FIG. 13 from the signal level of thelong signal existing on a specific position within the block, forexample the central position of the block is used for the synthesisprocessing of each pixel in the block. In this case, it is unnecessaryto determine the synthesis coefficient for each pixel, thereby allowingthe processing to be simplified. The position of the pixel used todetermine the synthesis coefficient is not required to be limited to thecentral position of the block.

In Embodiment 1 of the present invention, in the signal synthesizermeans 7, a configuration is also considered in which the synthesiscoefficient k is determined not from the long signal but from the shortsignal converted to a frame image.

A configuration is also considered in which the synthesis coefficient kis determined not from the short signal converted to a frame image butfrom the short signal as a field image. In this case, as can be seenfrom FIG. 12, the short signal corresponding to the even number line ofthe long signal does not exist, so that the synthesis coefficient cannotbe determined when the short signal is still in that condition. In thiscase, it is sufficient that the synthesis coefficient on the positioncorresponding to the even number line of the long signal is determinedfrom the peripheral short signal or the peripheral synthesiscoefficient.

In Embodiment 1 of the present invention, an example of a method ofdetermining the synthesis coefficient k from the signal level is shownin FIG. 13, but the method of determining the synthesis coefficient k isnot limited to such a method, and for example, a method is considered inwhich k is nonlinearly determined depending on the brightness level asshown, for example, in FIG. 37.

In Embodiment 2 of the present invention, the short signal is taken as aone-field image read in the field read mode, but is not limited to sucha configuration, and for example, a configuration is considered in whichthe signal is read by thinning out the horizontal line signal in thevertical direction, as shown in FIG. 33 by way of example. In this case,the short signal is read without mixing of charges accumulated on thetwo upper/lower photodiodes in the solid-state imaging element, therebymaking the two horizontal line addition processing unnecessary. In theinterpolation processing by the interpolation means 702 shown in FIG. 4,it is necessary to perform the interpolation processing in a manner tomatch the number of horizontal lines of the short signal with the longsignal. That is, in the interpolation means 702, a horizontal linesignal for two lines is produced by the interpolation processing betweenhorizontal line signals of the short signal. This causes the shortsignal and the long signal to have the same signal type, therebyallowing both the signals to be synthesized by the weighting additionmeans 703 shown in FIG. 4. However, the long signal to which the twoupper/lower lines are not added is supplied to the brightness signalextraction means 70401, so that it is necessary to add newly the twohorizontal line addition processing for extracting the brightness signalto the means. Or, it is necessary to provide the same means as the twoline addition means 701 on the pre-step of the brightness signalextraction means 70401 so that the long signal to which the twoupper/lower lines are added is supplied to the brightness signalextraction means 70401. When the short signal is read by thinning outthe horizontal line signal in this manner, the two horizontal lineaddition processing for the long signal is described to be unnecessary,but the processing is not limited to such a configuration, and aconfiguration is also considered in which the long signal and the shortsignal are subject to the two horizontal line addition processing andthen to the synthesis processing.

In Embodiment 2 of the present invention, the two signals having adifferent amount of exposure are taken as the short signal as one-fieldsignal and as the long signal as one-frame image, but are not limited tosuch a configuration, and may be taken as the long signal as one-fieldsignal and as the short signal as one-frame image depending on theapplication of the solid-state imaging element. In this case, it issufficient to provide a configuration in which as shown in FIG. 34, thelong signal is subject to the interpolation processing in the verticaldirection by the interpolation means 702, while the short signal issubject to the addition of the two upper/lower lines adjacent to eachother by the two horizontal line addition means 701, and in which thesynthesis coefficient used in the weighting addition means 703 isdetermined from the long signal after the interpolation processing. Aconfiguration is also considered in which the synthesis coefficient isdetermined from the long signal before the interpolation processing, inwhich case, it is sufficient that the corresponding long signal does notexist on the even-number lines of the short signal as shown in FIG. 34(a), and thus the synthesis coefficient k cannot be determined, so thatthe synthesis coefficient on the position of the even-number lines ofthe short signal is determined from the synthesis coefficient determinedfrom the horizontal line of the long signal existing on the sameposition as the upper/lower lines of the even-number lines of the shortsignal. In this manner, the synthesis coefficient is determined the longsignal as one-field image and the short signal as one-frame image,thereby allowing an expanded dynamic range image with a high resolutionat a high-brightness portion to be picked up.

In Embodiment 2 of the present invention, in the signal synthesizermeans 7, the synthesis coefficient k for synthesizing the long signaland the short signal is determined for each pixel of the long brightnesssignal, but the means is not limited to such a configuration, and forexample, a configuration is also considered in which the synthesiscoefficient k is determined for each pixel from the mean value, minimumvalue, maximum value or intermediate value of the brightness signallevel of a plurality of pixels, or in which of the values of kdetermined for each pixel, the mean value, minimum value, maximum valueor intermediate value of k determined from a plurality of pixels istaken as the synthesis coefficient for each pixel.

In Embodiment 2 of the present invention, in the signal synthesizermeans 7, a configuration is also considered in which a synthesiscoefficient for a block consisting of a plurality of pixels isdetermined to perform synthesization. For example, in FIG. 38, when thelong signal (Ye+Mg) L11 and (Cy+G) L12, and the long signal (Ye+Mg) L13and (Cy+G) L14 are taken as one block, respectively, while the shortsignal (Ye+Mg) S11 and (Cy+G) S12, and the short signal (Ye+Mg) S13 and(Cy+G)S14 existing on the same position as the long signals are taken asone block, respectively, it is possible to determine a synthesiscoefficient for each block in two pixel units and performsynthesization. At this time, for example, the synthesis of the blockconsisting of the long signal (Ye+Mg)L11 and (Cy+G)L12 and the blockconsisting of the short signal (Ye+Mg)S11 and (Cy+G)S12 is performed,when the synthesis coefficient of these blocks is taken as kb11, as inthe equation (18), respectively. In this case, it is sufficient that kdetermined by the method shown in FIG. 18 from either the signal levelof the long brightness signal (for example, (Ye+Mg)L11 and (Cy+G)L12)corresponding to the block, or at least one of the mean value, minimumvalue, maximum value and intermediate value of the long brightnesssignal (for example, YL11 and YL12 in FIG. 38) corresponding to theblock is taken as the synthesis coefficient kb11 of the block. Aconfiguration is also considered in which one of the mean value, minimumvalue, maximum value and intermediate value of the k values (forexample, k1, k2 in FIG. 38) for each pixel determined by the methodshown in FIG. 18 from each signal level of the long brightness signalcorresponding to the block is taken as the synthesis coefficient kb11 ofthe block. It is needless to say that the number of pixels in the blockis not limited to two pixels.

In Embodiment 2 of the present invention, in the signal synthesizermeans 7, a configuration is also considered in which a block consistingof a plurality of pixels is provided, and a synthesis coefficientdetermined by the method shown in FIG. 18 from the signal level of thelong signal existing on a specific position within the block, forexample the central position of the block is used for the synthesisprocessing of each pixel in the block. In this case, it is unnecessaryto determine the synthesis coefficient for each pixel, thereby allowingthe processing to be simplified. The position of the pixel used todetermine the synthesis coefficient is not required to be limited to thecentral position of the block.

In Embodiment 2 of the present invention, in the signal synthesizermeans 7, a configuration is also considered in which the synthesiscoefficient is determined not from the long brightness signal but fromthe brightness signal (short brightness signal) extracted from the shortsignal converted to the frame image. Also, a configuration is alsoconsidered in which the synthesis coefficient k is determined not fromthe brightness signal extracted from the short signal converted to theframe image, but from the brightness signal extracted from the shortsignal as the field image. In this case, as can be seen from FIG. 12,the short signal corresponding to the even-number line of the longsignal does not exist, so that when it is still in this condition, thesynthesis coefficient cannot be determined. In this case, it issufficient that the synthesis coefficient on the position correspondingto the even-number line of the long signal is determined from theperipheral short brightness signal or the peripheral synthesiscoefficient.

In Embodiment 2 of the present invention, an example of a method ofdetermining the synthesis coefficient k from the signal level is shownin FIG. 18, but the method of determining the synthesis coefficient k isnot limited to such a method, and for example, a method is considered inwhich k is nonlinearly determined depending on the brightness level asshown, for example, in FIG. 39.

In Embodiment 3 of the present invention, the short signal is taken as aone-field image read in the field read mode, but is not limited to sucha configuration, and a configuration is considered in which the signalis read by thinning out the horizontal line signal in the verticaldirection, as shown in FIG. 33 by way of example. In this case, theshort signal is read without mixing of charges accumulated on the twoupper/lower photodiodes in the solid-state imaging element, so that itis sufficient to provide a configuration as shown in, for example, FIG.40. In the configuration shown in FIG. 40, the mixing of the twoupper/lower pixels of the short signal is performed in the twohorizontal line adder means 27 (which is the same means as the twohorizontal line adder means 701, but is numbered as 27 to distinguish),and as a result, the same function/effect as the configuration shown inFIG. 20 can be realized. However, it is needless to say that in thebrightness signal interpolation means 12, the contents of theinterpolation processing varies with the thinning out the short signal.For example, when the short signal is a signal thinned out as shown inFIG. 30, as shown in, for example, FIG. 41, it is sufficient to produceinterpolated horizontal line signals for two lined between horizontallines of the short brightness signal (FIG. 41( c)). It is needless tosay that it is sufficient that similarly in the interpolation means 702,a required horizontal line signal is produced depending on the thinningout of the short signal. For the configuration in which the horizontalline signal is thinned out in the vertical direction to read as shown inFIG. 33, a configuration is shown in FIG. 40 which has twotwo-horizontal line adder means, that is, the two-horizontal line addermeans 701 and 27, but is not limited to such a configuration, and aconfiguration is also considered which has no two-horizontal line addermeans 701 and 27. In this case, including of means having the sameeffect as the two-horizontal line adder means 701 and 27 allows thebrightness extraction from the long signal and the short signal. Such aconfiguration having no two-horizontal line adder means 701 and 27 isalso valid in a pick up system in which the color filter formed on thesolid-state imaging element 3 consists of, for example, primary colorsof red (R), green (G) and blue (B), and the brightness signal and thecolor signal are obtained generally without mixing the charges on thephotodiode of the solid-state imaging element 3.

In Embodiments 3 and 4 of the present invention, the two signals havinga different amount of exposure are taken as the short signal asone-field image and as the long signal as one-frame image, but are notlimited to such a configuration, and may be taken as the long signal asone-field signal and as the short signal as one-frame image depending onthe application of the solid-state imaging device. In this case, it issufficient that the long brightness signal obtained from the long signalis subject to the interpolation processing in the vertical direction bythe brightness signal interpolation means 12, and the long signal afterthe interpolation processing and the short signal are synthesized in thebrightness signal synthesizer means 13, and that the synthesiscoefficient used at that time is determined from the long signal afterthe interpolation processing. As with Embodiments 1 and 2 of the presentinvention, a configuration is also considered in which the synthesiscoefficient is determined from the long signal before the interpolationprocessing. In this manner, the synthesis coefficient is determined fromthe long signal as one-field image and the short signal as one-frameimage, thereby allowing an expanded dynamic range image with a highresolution at a high-brightness part to be picked up.

In Embodiment 3 of the present invention, the interpolation means 720uses two one-line memories and performs the interpolation processingfrom the signal for two horizontal lines, but is not limited to suchconfiguration, and for example, a configuration is also considered inwhich the means uses a large number of one-line memories and performsthe interpolation processing by a high-rank interpolation processingfrom a large number of horizontal line signals. A configuration is alsoconsidered in which outputting repeatedly twice the one-horizontal linesinputted causes the number of horizontal lines to become double, thatis, the so-called previous-value interpolation to be performed.

In Embodiments 3 and 4 of the present invention, the brightness signalinterpolation means 12 takes the addition mean value of the twohorizontal line signal as the interpolation signal, but is not limitedto such a configuration, and a configuration is also considered in whichthe interpolation processing is performed by a high-rank interpolationprocessing from a large number of horizontal line signals, or in whichthe interpolation signal is obtained by a previous-value interpolation.

In Embodiments 3 and 4 of the present invention, in the brightnesssignal synthesizer means 13, the synthesis coefficient k forsynthesizing the long brightness signal and the short brightness signalis determined for each pixel of the long brightness signal, but themeans is not limited to such a configuration, and for example, aconfiguration is also considered in which the synthesis coefficient k isdetermined for each pixel from the mean value, minimum value, maximumvalue or intermediate value of the long brightness signal level of aplurality of pixels, or in which of the values of k determined for eachpixel, the mean value, minimum value, maximum value or intermediatevalue of k determined from a plurality of pixels is taken as thesynthesis coefficient for each pixel.

In Embodiments 3 and 4 of the present invention, in the brightnesssignal synthesizer means 13, a configuration is also considered in whicha synthesis coefficient for a block consisting of a plurality of pixelsis determined to perform synthesization. For example, in FIG. 43, whenthe long brightness signal YL11 and YL12, and YL13 and YL14 are taken asone block, respectively, while the short brightness signal YS11 andYS12, and YS13 and YS14 existing on the same position as the longbrightness signals are taken as one block, respectively, it is possibleto determine a synthesis coefficient for each block in two pixel unitsand perform synthesization. At this time, for example, the synthesis ofthe block consisting of the long brightness signal YL11 and YL12 and theblock consisting of the short brightness signal YS11 and YS12 isperformed, when the synthesis coefficient of these blocks is taken askb11, as in the equation (19). (YM is a brightness signal aftersynthesis)

YM11=(1−kb11)×YL11+kb11×YS11×D

YM12=(1−kb11)×YL12+kb11×YS12×D  (19)

In this case, it is sufficient that k determined by the method shown inFIG. 18 from either the signal level of the long brightness signal (forexample, Y11 and Y12 in FIG. 43) corresponding to the block, or at leastone of the mean value, minimum value, maximum value and intermediatevalue of the long brightness signal corresponding to the block is takenas the synthesis coefficient kb11 of the block. A configuration is alsoconsidered in which one of the mean value, minimum value, maximum valueand intermediate value of the k values (for example, k1, k2 in FIG. 43)for each pixel determined by the method shown in FIG. 18 from eachsignal level of the long brightness signal corresponding to the block istaken as the synthesis coefficient kb11 of the block. It is needless tosay that the number of pixels in the block is not limited to two pixels.

In Embodiments 3 and 4 of the present invention, in the brightnesssignal synthesizer means 13, a configuration is also considered in whicha block consisting of a plurality of pixels is provided, and a synthesiscoefficient determined by the method shown in FIG. 18 from the signallevel of the long brightness signal corresponding to a specific positionwithin the block, for example the central position of the block is usedfor the synthesis processing of each pixel in the block. In this case,it is unnecessary to determine the synthesis coefficient for each pixel,thereby allowing the processing to be simplified. The position of thepixel used to determine the synthesis coefficient is not required to belimited to the central position of the block.

In Embodiments 3 and 4 of the present invention, in the signalsynthesizer means 14 and 23, the synthesis coefficient k forsynthesizing the long signal and the short signal uses a valuedetermined for each pixel by the synthesis coefficient generation means1301 from the long brightness signal, but the determination is notlimited to such a configuration, and for example, a configuration isalso considered in which the synthesis coefficient generation means 1404is independently provided within the signal synthesizer means 14 and 23as shown in FIG. 42, and the synthesis coefficient k is determined foreach pixel from the mean value, minimum value, maximum value orintermediate value of the brightness signal level of a plurality ofpixels, or in which of the values of k determined for each pixel, themean value, minimum value, maximum value or intermediate value of kdetermined from a plurality of pixels is taken as the synthesiscoefficient for each pixel. Here, the function of the synthesiscoefficient generation means 1404 is the same as the synthesiscoefficient generation means 1301.

In Embodiments 3 and 4 of the present invention, in the signalsynthesizer means 14 and 23, a configuration is also considered in whicha synthesis coefficient for a block consisting of a plurality of pixelsis determined to perform synthesization. For example, in FIG. 38, whenthe long signal (Ye+Mg)L11 and (Cy+G)L12, and the long signal (Ye+Mg)L13and (Cy+G)L14 are taken as one block, respectively, while the shortsignal (Ye+Mg)S11 and (Cy+G)S12, and the short signal (Ye+Mg)S13 and(Cy+G)S14 existing on the same position as the long signals are taken asone block, respectively, it is possible to determine a synthesiscoefficient for each block in two pixel units and performsynthesization. At this time, for example, the synthesis of the blockconsisting of the long signal (Ye+Mg)L11 and (Cy+G)L12 and the blockconsisting of the short signal (Ye+Mg)S11 and (Cy+G)S12 is performed,when the synthesis coefficient of these blocks is taken as kb11, as inthe equation (18). In this case, it is sufficient that k determined bythe method shown in FIG. 18 from either the signal level of the longbrightness signal (for example, YL11 and YL12 in FIG. 38) existingspatially on the same position as respective blocks, or at least one ofthe mean value, minimum value, maximum value and intermediate value ofthe long brightness signal existing spatially on the same position asrespective blocks is taken as the synthesis coefficient kb11 of theblock. A configuration is also considered in which one of the meanvalue, minimum value, maximum value and intermediate value of the kvalues (for example, k1, k2 in FIG. 36) for each pixel determined by themethod shown in FIG. 18 from each signal level of the long brightnesssignal existing spatially on the same position as respective blocks istaken as the synthesis coefficient kb11 of the block. It is needless tosay that the number of pixels in the block is not limited to two pixels.

In Embodiments 3 and 4 of the present invention, in the signalsynthesizer means 14 and 23, a configuration is also considered in whicha block consisting of a plurality of pixels is provided, and a synthesiscoefficient determined by the method shown in FIG. 18 from the signallevel of the long brightness signal existing on a specific positionwithin the block, for example, the central position of the block is usedfor the synthesis processing of each pixel in the block. In this case,it is unnecessary to determine the synthesis coefficient for each pixel,thereby allowing the processing to be simplified. The position of thepixel used to determine the synthesis coefficient is not required to belimited to the central position of the block.

In Embodiments 3 and 4 of the present invention, a configuration is alsoconsidered in which the synthesis coefficient k used in the signalsynthesizer means 14 and 23 is taken as a value obtained bymultiplication of the value obtained by the above-mentioned method fromthe long brightness signal by a certain coefficient, or a value obtainedby addition/subtraction of a certain coefficient.

In Embodiments 3 and 4 of the present invention, in the brightnesssignal synthesizer means 13, and signal synthesizer means 14 and 23, aconfiguration is also considered in which the synthesis coefficient k isdetermined not from the long brightness signal but from the brightnesssignal (short brightness signal) extracted from the short signalconverted to the frame image. Also, a configuration is also consideredin which the synthesis coefficient k is determined not from thebrightness signal extracted from the short signal converted to the frameimage, but from the brightness signal extracted from the short signal asthe field image. In this case, as can be seen from FIG. 12, the shortsignal corresponding to the even-number line of the long signal does notexist, so that when it is still in this condition, the synthesiscoefficient cannot be determined. In this case, a method is consideredin which the synthesis coefficient on the position corresponding to theeven-number line of the long signal uses the peripheral short brightnesssignal or the peripheral synthesis coefficient as they are, or in whichthe synthesis coefficient is determined from the mean value, maximumvalue, minimum value of intermediate value of the peripheral synthesiscoefficient.

In Embodiments 3 and 4 of the present invention, an example of a methodof determining the synthesis coefficient k from the brightness signallevel is shown in FIG. 18, but the method of determining the synthesiscoefficient k is not limited to such a method, and for example, a methodis considered in which k is nonlinearly determined depending on thebrightness level as shown, for example, in FIG. 39.

In Embodiment 4 of the present invention, the short signal is taken as aone-field image read in the field read mode, but is not limited to sucha configuration, and a configuration is considered in which the signalis read by thinning out the horizontal line signal in the verticaldirection, as shown in FIG. 33 by way of example. In this case, theshort signal is read without mixing of charges accumulated on the twoupper/lower photodiodes in the solid-state imaging element, so that, forexample, when the two upper/lower pixels is to be mixed by the twohorizontal line addition means as with FIG. 40, with the result that thesame function/effect as the configuration shown in FIG. 30 is realized.However, it is needless to say that in the brightness signalinterpolation means 12 as with Embodiment 3 of the present invention,the contents of the interpolation processing varies with the thinningout of the short signal. It is needless to say that similarly also inthe thinning out means 22, it is sufficient that thinning out isperformed depending on the thinning out of the short signal in a mannerthat the long signal exhibits the same signal type as the short signal.

In Embodiment 4 of the present invention, there has been explained aconfiguration in which the thinning out processing of the long signal inthe vertical direction is performed by the thinning out means 22, but aconfiguration is also considered in which there is provided ahorizontal-direction thinning out means 27 having a function of thinningthe pixel in the horizontal direction out of said image signal as shownin FIG. 44, whereby the pixels in the horizontal direction of both thelong signal and the short signal passing through the two horizontal lineadder means 701 is thinned out to, for example, a half. In this case,thinning out the pixels in the horizontal direction to a half asmentioned above allows the one-line memories 15, 16 for synchronizationto be substituted by half-line memories with a capacity a half thereof.Thinning out the pixels in the horizontal direction in this mannerallows the solid-state imaging device to be further simplified inconfiguration and to be made cheaper. In this case, when the long signaland the short signal are previously subject to the band restriction inthe horizontal direction before the thinning out processing in thehorizontal direction is performed, an unwanted reflection does not occurby the thinning out processing. It is needless to say that when thesignals are also subject to the band restriction in the verticaldirection in the same manner, an unwanted reflection can be avoided evenin performing the thinning out processing in the vertical direction.

In all of the above-mentioned embodiments of the present invention, thelong signal and the short signal are stored in the image memory 6 for atime, but the procedure is not limited to such a method, and a method isalso considered in which, for example, only one of either the longsignal or the short signal is stored in the image memory 6, and the readof the remaining signal from the solid-state imaging element 3 and theread of the signal from the image memory 6 are synchronized to performthe synthesis processing. In this case, the capacity of the image memory6 can be reduced, and the solid-state imaging device can be configuredat a cheaper cost.

In all of the above-mentioned embodiments of the present invention, thearrangement of the color filters formed on the solid-state imagingelement 3 is explained using complementary-color checkered typeconsisting of four colors of magenta, green, yellow and cyan as shown inFIG. 3, but is not limited to such a configuration, and by way ofexample, an arrangement is also considered in which magenta (Mg) andgreen (G) are not reversed in position for each line as shown in FIG.45, and a configuration is also considered in which twocomplementary-color filters for green (G) and cyan (Cy), yellow (Ye) arearranged in stripe shape as shown in FIG. 46.

In all of the above-mentioned embodiments of the present invention, thearrangement of the color filters formed on the solid-state imagingelement 3 is explained using configuration consisting of four colors ofmagenta, green, yellow and cyan as shown in FIG. 3, but is not limitedto such a configuration, and a configuration is also considered whichuses filters for primary colors consisting of green (G), blue (B) andred (R). By way of example as the filter arrangement, there areconsidered the bayer method shown in FIG. 47, the inter-line methodshown in FIG. 48, the G stripe RB completely checkered method shown inFIG. 49, the stripe method shown in FIG. 50, the diagonal stripe methodshown in FIG. 51, the G stripe RB line sequential method shown in FIG.52, the G stripe RB point sequential method shown in FIG. 53, and thelike. It is needless to say that when the primary color filter is used,the brightness signal is determined according to the equation (20).

Brightness signal=0.3×R+0.59×G+0.11×B  (20)

In all of the above-mentioned embodiments of the present invention, thearrangement of the color filters formed on the solid-state imagingelement 3 is explained using complementary-color checkered typeconsisting of four colors of magenta, green, yellow and cyan as shown inFIG. 3, and further, the read of the short signal is explained as thefield read, so that the two horizontal upper/lower line additionprocessing of the long signal by the two horizontal line adder means 701is included to match in signal type the long signal with the shortsignal, but are not limited to such a configuration, and it is needlessto say that when another filter arrangement is employed as shown inFIGS. 45 to 53, or when rather than the field read, the thinning outread as shown in FIG. 33 is performed, the two horizontal upper/lowerline addition processing is not necessarily required.

In all of the above-mentioned embodiments of the present invention, aconfiguration is also considered in which thresholds Th_max, Th_min,Th_max′, and Th_min′ in determining the synthesis coefficient are set asin the equation (21), respectively, and the long-time exposure signaland the short-time exposure signal are switched by a signal level ratherthan the weighting addition.

Th_max=Th_min

Th_max′=XTh_min′  (21)

(4) Embodiment 5

The fifth embodiment is wherein the image synthesizer means 38 has thesame means as the signal synthesizer means included in Embodiments 1 to4, and in addition, an exposure amount ratio detector means 36 isincluded.

That is, when in a method of using a plurality of image signals having adifferent exposure amount and synthesizing the signals to pick up animage with an expanded dynamic range, Embodiment 5 of the presentinvention has a configuration in which the exposure time is controlledusing a mechanical shutter serving also as an optical aperture, or whena subject is picked up in a such condition of light source illuminationthat the brightness level of an illumination for the subject, forexample, a fluorescent lamp varies periodically, even when the ratio inthe exposure amount of a plurality of image signals having a differentexposure amount is not determined simply as the ratio in the exposureamount, an synthesis image after the image synthesis processing hasstably continuous gradation characteristics by actually determining ofthe exposure amount ratio in the full screen field in each pixel of thelong-time exposure signal and the short-time exposure signal.

FIG. 54 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 5 of the present invention. In FIG. 54, thenumeral 31 designates a mechanical shutter serving also as an opticalaperture; and the numeral 32 designates a solid-state imaging element,and is taken as a whole pixel read-type CCD (Charge Coupled Device) inEmbodiment 5. The numeral 33 designates two horizontal line adder meansfor adding two horizontal lines to the output from the solid-stateimaging element 32; the numeral 34, an image memory for storing thesignal for two frames after addition of the horizontal lines from thesolid-state imaging element 32; the numeral 35, a shutter drive controlmeans for performing the open/close control of the mechanical shutter31; the numeral 36, an exposure amount ratio detector means fordetermining the exposure amount ratio of a plurality of image signalshaving a different exposure amount; the numeral 37, a gain adjustmentmeans for performing the gain adjustment to a plurality of image signalshaving a different exposure amount; and the numeral 38, imagesynthesizer means for performing the image synthesization to expand thedynamic range.

In the block of the exposure amount ratio detector means 36, the numeral361 designates signal level determination means; the numeral 362,integrator means for the long signal; the numeral 363, integrator meansfor the short signal; and the numeral 364, exposure amount ratiooperation means for determining the ratio (LSUM/SSUM) of an integratedvalue LSUM in the whole screen field by the long signal integrator means362 to an integrated value SSUM in the whole screen field by the shortsignal integrator means 363.

FIG. 55 is a block diagram showing a configuration example of the twohorizontal line adder means 33 in block diagram of FIG. 54. In thediagram, the numeral 331 designates a one-line memory which is means fordelaying by one horizontal synchronous period said image signal forone-line memory outputted from the solid-state imaging element 32. Thenumeral 332 designates an adder in which the horizontal line signaldelayed in the one-line memory 331 and the horizontal line signalinputted into the two horizontal line adder means 33 are added to eachother, whereby the addition of two upper/lower lines adjacent to eachother is performed.

FIG. 56 shows a configuration example of the image memory 34 in theblock diagram of FIG. 54. The image memory 34 has a configuration inwhich the image memory 34 requires a memory capacity capable of storingtwo frames, and for example, as shown in FIG. 56, has two frame memoriesof a long signal frame memory 341 a and a short signal frame memory 341b, so that the long signal and the short signal can be controlled in amanner that they are stored independently of respective frame memories.

With respect to the solid-state imaging device of the present fifthembodiment configured as described above, the operation thereof will beexplained hereinafter.

A concrete example will be explained of a case where Embodiment 5 picksup two images of the short-time exposure signal (short signal) and thelong-time exposure signal (long signal) to synthesize these signals.

First, the method of picking up the short signal and the long signalwill be explained using FIG. 57. FIG. 57 is a timing chart with respectto the exposure of a subject image, to the read of an exposed signal,and to the read/write operation of the image memory 34 in thesolid-state imaging device 32. In FIG. 57, the numeral 3401 designates atiming of the synchronous signal in the vertical direction; the numeral3402, a timing of a read control pulse for controlling the signal chargeread from the photodiode of the solid-state imaging element 32; thenumeral 3403, a timing showing the opening/closing state of themechanical shutter 31; the numeral 3404, a timing showing the exposuresignal amount on the photodiode of the solid-state imaging element 32;the numeral 3405, a timing of the signal outputted from the solid-stateimaging element 32; the numeral 3406, a timing of the input (write)signal of the image memory 34; the numeral 3407, a timing of the output(read) signal from the short signal storage means (the short signalframe memory 341 b) of the image memory 34; and the numeral 3408, atiming of the output (read) signal from the long signal storage means(the long signal frame memory 341 a) of the image memory 34.

At the short signal exposure, with the mechanical shutter 31 beingopened, using an electrical shutter function, the exposure is performedfor a required exposure time, for example, one thousandth of a second.After the exposure for one thousandth of a second is finished, thecharge accumulated on the photodiode is moved to the vertical transferCCD by the read control pulse. At this time, the read mode of thesolid-state imaging element 32 should drive in the whole pixel readmode.

Then, after short signal is moved to the vertical transfer CCD, the longsignal should be exposed. The exposure time of the long signal is takenas, for example, one hundredth of a second. The exposure time of thelong signal should be controlled by the opening/closing of themechanical shutter 2. Although concurrently with the above-mentionedexposure, the short signal for one frame is outputted from thesolid-state imaging element 32, the charge accumulated on the photodiodefor the long signal is moved to the vertical transfer CCD by the readcontrol pulse. At this time, the read mode of the solid-state imagingelement 32 should drive in the read mode. The cycle of the verticalsynchronous signal is taken as, for example, three tenths of a second,and the signal read for one field should be completed within one cycleof the vertical synchronous signal.

After the two signals of the short signal and the long signal with adifferent exposure time obtained by the solid-state imaging element 32are respectively and individually subject to the two horizontal lineaddition processing for adding/mixing the two upper/lower line signalsadjacent to each other on the solid-state imaging element 32 by the twohorizontal line adder means 33, as shown in the timing of the numeral3406, the short signal is for a time stored in the short signal framememory 341 b of the image memory 34 during period {circle around (1)},while the long signal is for a time stored in the long signal framememory 341 a of the image memory 34 during period {circle around (2)}.

Then, the short signal and the long signal are read from the imagememory 34 during periods {circle around (3)}, {circle around (4)} asshown in the timing of the numerals 3407 and 3408.

During period {circle around (3)}, in the exposure amount ratio detectormeans 36, the exposure amounts of each pixel in the respective wholescreen field of the short signal (short frame signal) and the longsignal (long frame signal) are integrated, whereby the whole amount ofexposure (SSUM, LSUM) is determined.

During period {circle around (3)}, the timing of reading the shortsignal and the long signal from the image memory 34 is such that theshort signal and the long signal are read sequentially from the firstline in a manner that the lines on the positions corresponding to eachother on the solid-state imaging element 32 are outputted in thattiming. Here, during period {circle around (3)}, the short signal andthe long signal read from the image memory 34 are inputted into theexposure amount ratio detector means 36. The exposure amount ratiodetector means 36 performs the exposure amount ratio detection in thewhole screen of the short signal and the long signal, so that anintegrated value of the exposure amount of each pixel in the respectivewhole screen field of the short signal and the long signal isdetermined, and then the ratio of the integrated value (LSUM/SSUM) isdetermined.

The short signal and the long signal inputted in synchronism with theline into the exposure amount ratio detector means 36 are determined,respectively, for the level in signal level determination means 361, anda signal determined to be at a level out of a predetermined range isneglected so as not to be integrated in a long signal integrator means362 and a short signal integrator means 363. Here, the level out of apredetermined range means that it is a level at which the ratio in theexposure amount is difficult to determine, and more specifically, thatwith respect to signal, the long signal is a signal of a high brightnessportion near the saturation level of the solid-state imaging element 32,while short signal is a signal of a low brightness portion on which thenoise effect largely acts.

FIG. 58 is a block diagram showing a configuration in the signal leveldetermination means 361. Here, in FIG. 58, the numeral 36101 designatesmeans (a comparator A) for determining the level of the long signal; thenumeral 36102, means (a comparator B) for determining the level of theshort signal; the numeral 36103, an OR gate; the numeral 36104, a gatemeans for gating the long signal; and the numeral 36105, a gate meansfor gating the short signal. In the means 36101 for determining thelevel of the long signal, the gate means 36104 gates the high-brightnesssignal near the saturation level of the solid-state imaging element 32so as not to be integrated by the long signal integrator means 362,while the gate means 360105 gates also the short signal of the pixelcorresponding thereto so as not to be integrated by the short signalintegrator means 363.

In the means 36102 for determining the level of the short signal, thegate means 36105 gates the low-brightness signal on which the noiseeffect largely acts so as not to be integrated by the short signalintegrator means 363, while the gate means 360104 gates also the longsignal of the pixel corresponding thereto so as not to be integrated bythe long signal integrator means 362.

The integrated value LSUM as a result of integration in the long signalintegrator means 362 in the whole field of one screen, and theintegrated value SSUM as a result of integration in the long signalintegrator means 363 are inputted into exposure amount ratio operationmeans 364, and an operation shown in the following equation (22) isperformed to determine an exposure amount ratio D in the whole screenfield.

D=LSUM/SSUM  (22)

The above-mentioned method allows the exposure amount ratio D in theaverage of the whole screen of the exposure amount of the short signalcontrolled in the exposure amount by an electronic shutter to theexposure amount of the long signal controlled in the exposure amount bythe mechanical shutter to be extremely correctly obtained.

Also when the illumination of a subject is a fluorescent lamp, it ispossible to obtain the exposure amount ratio of the short signal to thelong signal by the completely same method and timing as theabove-mentioned explanation, so that the explanation in this case willbe omitted.

Then, during period {circle around (4)} shown in FIG. 57, as with period{circle around (3)}, the long signal and the short signal are read fromthe image memory 34, and the short signal is multiplied in the gainadjustment means 37 by a D-fold gain equivalent to the exposure amountratio operated during period {circle around (3)}.

A signal after the D-fold gain is given to the short signal is expressedas the short′ signal hereinafter. Based on the long signal read from theimage memory 34 and on the short′ signal whose gain is adjusted, asynthesization is performed by the image synthesizer means 38 at apost-stage with a method as shown in the well-known principle of thedynamic range expansion.

In the above-mentioned manner, the ratio in the exposure amount of thelong signal to the short signal can be determined at a high accuracy, sothat the synthesis image by the image synthesizer means 38 using thelong signal and the gain-adjusted short signal can obtain gradationcharacteristics which are continuously stable from a low-brightnessportion to a high-brightness portion.

The signal synthesized in the image synthesizer means 38 is subject toprocessings such as separating of brightness signal from color signal,noise removing, edge enhancing, gamma correcting, matrix operating, andencoding to a specific format, in the later processing. The processingin the image synthesizer means 38 and a processing which is laterperformed are not directly related to the object of the presentinvention, so that a detailed explanation will be omitted.

Although in Embodiment 5 of the present invention, the solid-stateimaging element 32 is explained using a whole pixel read-type CCD(Charge Coupled Device), the present invention is not limited to such aCCD, but can be applied to an imaging means capable of outputting aplurality of image signals having a variously different exposure amount,such as means for performing frame read, and means for performing twoline addition/read on the solid-state imaging element.

Although in Embodiment 5 of the present invention, with respect to asignal by which two lines are added to the output from the solid-stateimaging element 32 by the two horizontal line adder means 33, theexposure ratio of the long signal to the short signal is detected in theexposure amount ratio detector means 36, the present invention is notlimited to such a method, and there is no particular problem even whenwith respect to a signal by which two lines are not added to the outputfrom the solid-state imaging element 32, the exposure ratio is detected.

Although in Embodiment 5 of the present invention, the image memory 34is capable of store two frames, the present invention is not limited tosuch a capacity, and the image memory 34 may be capable of storing aplurality of output signals having a different exposure amount from theimaging means.

Although in Embodiment 5 of the present invention, both the long signaland the short signal having a different exposure amount outputted fromthe solid-state imaging element 32 are frame signals, the presentinvention is not limited to such signals, but can be applied to a casewhere both are field signals, or where the number of lines is differentfor each signal having a different exposure amount such that one isfield, and the other is frame. When the number of lines is different foreach signal having a different exposure amount as described above, theexposure amount ratio can be detected by controlling the image memory 34during exposure amount ratio detection, or by making the pixel at acorresponding position output to the exposure amount ratio detectormeans 36 in the same timing in a manner that a signal having a largenumber of lines is matched with the one having a small number of lines.

(5) Embodiment 6

In the case of Embodiment 5, when the long-time exposure signal and theshort-time exposure signal which are different in exposure amount fromeach other are used and synthesized to pick up an image with an expandeddynamic range, in order to obtain stably continuous gradationcharacteristics as a synthesis image after the image synthesisprocessing, an exposure amount ratio is actually determined in the wholescreen field of the long-time exposure signal and the short-timeexposure signal, and using a signal to which a gain equivalent to theexposure amount ratio is given, and the long-time exposure signal, theimage synthesis processing is performed.

At this time, it takes a time equivalent to one frame time to detect theexposure amount ratio of the long-time exposure signal and theshort-time exposure signal, and further it takes a time equivalent toone frame time to synthesize the long-time exposure signal and theshort-time exposure signal, so that the processing time in total tendsto become longer.

The sixth embodiment, in the configuration in Embodiment 5, said imagesignal from the imaging means and said image signal from the imagememory are switched to output to the exposure amount ratio detectormeans, whereby it is intended to reduce the time required from thepicking up start at the imaging means described in Embodiment 5 to theimage synthesis processing end.

FIG. 59 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 6 of the present invention. In FIG. 59, theconfiguration is exactly the same as Embodiment 5 other than thecomponent designated at the numeral 39, so that the explanation will beomitted here. The numeral 39 in FIG. 59 designates image signalswitching means for switching between a signal after the output of thesolid-state imaging element 32 is subject to the addition processing atthe two horizontal line adder means 33 (taken as the side A), and anoutput signal from the long signal storing means of the image memory 34(taken as the side B). That is, said image signal switching means 39switches the long signal, when sent to the exposure amount ratiodetector means 36, to the side A to directly output the long signal fromthe two horizontal line adder means 33 to the exposure amount ratiodetector means 36, and switches the long signal, when sent to the imagesynthesizer means 38, to the side B to output the long signal from theimage memory 34 to the image synthesizer means 38.

With respect to the solid-state imaging device of the present secondembodiment configured as described above, the operation thereof will beexplained hereinafter.

A concrete example will be explained of a case where Embodiment 6 picksup two images of the short-time exposure signal (short signal) and thelong-time exposure signal (long signal) to synthesize these signals.

First, the method of picking up the short signal and the long signalwill be explained using FIG. 60. FIG. 60 is a timing chart with respectto the exposure of a subject image, to the read of an exposed signal,and to the read/write operation of the image memory 34 in thesolid-state imaging device 32. In FIG. 60, the numeral 31101 designatesa timing of the synchronous signal in the vertical direction; thenumeral 31102, a timing of a read control pulse for controlling thesignal charge read from the photodiode of the solid-state imagingelement 32; the numeral 31103, a timing showing the opening/closingstate of the mechanical shutter 31; the numeral 31104, a timing showingthe exposure signal amount on the photodiode of the solid-state imagingelement 32; the numeral 31105, a timing of the signal outputted from thesolid-state imaging element 32; the numeral 31106, a timing of the input(write) signal of the image memory 34; the numeral 31107, a timing ofthe output (read) signal from the short signal storage means (the shortsignal frame memory 341 b) of the image memory 34; and the numeral31108, a timing of the output (read) signal from the long signal storagemeans (the long signal frame memory 341 a) of the image memory 34.

At the short signal exposure, with the mechanical shutter 31 beingopened, using an electrical shutter function, the exposure is performedfor a required exposure time, for example, one thousandth of a second.After the exposure for one thousandth of a second is finished, thecharge accumulated on the photodiode is moved to the vertical transferCCD by the read control pulse. At this time, the read mode of thesolid-state imaging element 32 should drive in the whole pixel readmode.

Then, after short signal is moved to the vertical transfer CCD, the longsignal should be exposed. The exposure time of the long signal is takenas, for example, one hundredth of a second. The exposure time of thelong signal should be controlled by the opening/closing of themechanical shutter 31. Although concurrently with the above-mentionedexposure, the short signal for one frame is outputted from thesolid-state imaging element 32, the charge accumulated on the photodiodefor the long signal is moved to the vertical transfer CCD by the readcontrol pulse. At this time, the read mode of the solid-state imagingelement 32 should drive in the read mode. The cycle of the verticalsynchronous signal is taken as, for example, one thirtieths of a second,and the signal read for one field should be completed within one cycleof the vertical synchronous signal.

After the two signals of the short signal and the long signal having adifferent exposure time obtained by the solid-state imaging element 32are respectively and individually subject to the two horizontal lineaddition processing for adding/mixing the two upper/lower line signalsadjacent to each other on the solid-state imaging element 32 by the twohorizontal line adder means 33, as shown in the timing of the numeral31106, the short signal is for a time stored in the short signal framememory 341 b of the image memory 34 during period {circle around (1)},while the long signal is for a time stored in the long signal framememory 341 a of the image memory 34 during period {circle around (2)}.

During period {circle around (2)}, by switching said image signalswitching means 39 to the side A, the long signal being forcedlyinputted into the long signal storage means of the image memory 34 isinputted into the signal level determination means 361 in the exposureamount ratio detector means 36, and at the same time, as shown in thetiming of the numeral 31107, the short signal stored in the short signalstorage means of the image memory 34 is read and then inputted into thesignal level determination means 361 in the exposure amount ratiodetector means 36. In this case, with respect to the long signalinputted through said image signal switching means 39 and to the shortsignal outputted from the image memory 34, the long signal and the shortsignal of the lines on the respective positions on the solid-stateimaging element 32 are controlled to be inputted into the signal leveldetermination means 361 in the exposure amount ratio detector means 36at the same timing, whereby during period {circle around (2)}, theprocessing of the exposure amount detection is performed. The processingof the exposure amount detection during period {circle around (2)}conforms to the processing of the exposure amount detection duringperiod {circle around (3)} in Embodiment 5, so that the explanation willbe omitted here. It is wherein the processing during period {circlearound (2)} in Embodiment 6 precedes that in Embodiment 5 by one frameperiod.

Then, during period {circle around (3)} shown in FIG. 60, the longsignal and the short signal are read from the image memory 34. Duringperiod {circle around (3)}, by switching said image signal switchingmeans 39 to the side B, the long signal from the long signal storagemeans of the image memory 34 is inputted into the image synthesizermeans 38, and at the same time, the short signal from the short signalstorage means of the image memory 34 is multiplied by a D-fold gainequivalent to the exposure amount ratio D operated during period {circlearound (2)} in the gain adjustment means 37, and then inputted into theimage synthesizer means 38.

A signal after the D-fold gain is given to the short signal is expressedas the short′ signal hereinafter. As with Embodiment 5, based on thelong signal read from the image memory 34 and on the short′ signal whosegain is adjusted, a synthesization is performed by the image synthesizermeans 38 at a post-stage with a method as shown in the well-knownprinciple of the dynamic range expansion.

As shown above, the addition of said image signal switching means forswitching between said image signal from the imaging means and saidimage signal from the image memory allows the time required from thepicking up start at the imaging means described in Embodiment 5 to theimage synthesis processing end to be reduced by one frame time.

(6) Embodiment 7

When a subject is picked up under different light source such as in theoutdoors or indoors, by a method of using the short-time exposure signaland the long-time exposure signal having a different exposure amount andsynthesizing these signals to pick up an image with an expanded dynamicrange, there is a difference in the light aspect in that the brightnesslevel of the subject varies periodically due to a fluorescent lamp inthe indoors, while the brightness level of the subject does not varyperiodically due to the sunlight in the outdoors. Hence, for the subjectunder a plurality of light sources, the exposure amount ratio of thelong-time exposure signal to the short-time exposure signal is differentfor each portion of the image, so that it is impossible to give a gainequivalent to an uniform exposure amount ratio in the whole screen fieldas described in Embodiment 5 to the short-time exposure signal.

In Embodiment 7, even when a plurality of light sources exist, ansynthesis image after the image synthesis processing to expand thedynamic range has stably continuous gradation characteristics bydividing of the screen into a plurality of blocks, and by determining ofthe exposure amount ratio in the full screen field for each pixelbetween the long-time exposure signal (long signal) and the short-timeexposure signal (short signal).

FIG. 61 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 7 of the present invention. In FIG. 61, theconfiguration is exactly the same as Embodiment 5 other than theinternal configuration of the exposure amount ratio detector means 36,so that the explanation will be omitted here. In the exposure amountratio detector means 36 of FIG. 61, the numeral 361 designates signallevel determination means; the numerals 365 a, 365 b, multiplexers; thenumerals 366 a, 366 b, selectors; the numeral 367, a long signalintegrator means for dividing the screen into n blocks to determine anintegrated value LSUM of the long signal for each block; and the numeral368, a short signal integrator means for dividing the screen into nblocks to determine an integrated value SSUM of the short signal foreach block. In the long signal integrator means 367 and the short signalintegrator means 368 in block units, symbols ΣBL1, ΣBL2 through ΣBLndesignate integrator means in n block units. The numeral 364 designatesexposure amount ratio operation means for receiving the integrated valueLSUM for each block by the long signal integrator means 367 or theintegrated value SSUM for respective block by the short signalintegrator means 368 obtained by switching between these integratedvalues by the selectors 366 a, 366 b to determine the ratio of theintegrated value LSUM to the integrated value SSUM in block units.

With respect to the solid-state imaging device of the present seventhembodiment configured as described above, the operation thereof will beexplained hereinafter.

First, with respect to the method of picking up the short signal and thelong signal and to the method of controlling the exposure time and thesolid-state imaging element 32 and the image memory 34, the concreteexample in Embodiment 7 is exactly the same as that described in fifthembodiment, so that the explanation will be omitted here. The timing ofthe output signal from the solid-state imaging element 32 and the timingof the input/output signal into/from the image memory 34 at that timeare exactly the same as the timing chart of FIG. 57 used to explainEmbodiment 5, so that Embodiment 7 will be explained also using FIG. 57.

In the concrete example of Embodiment 7, there will be explained a casewhere the exposure amount ratio is determined for each block obtained bydividing the screen into 8×6=48 blocks as shown in FIG. 62. As anexample of the difference in the light source aspect, the fields of theblock numbers 6, 7, 8, 14, 15, 16, 22, 23, 24 are subject fields inwhich the outdoor sunlight exhibits the light source, and fields otherthan the above fields are subject fields in which the fluorescent lampexhibits the light source.

In Embodiment 7, during periods {circle around (1)}, {circle around (2)}shown in FIG. 57, the signal outputted from the solid-state imagingelement 32 is subject to the two horizontal lines addition processing bythe two horizontal line adder means 33, and then is stored for a time inthe image memory 34.

During period {circle around (3)}, the long signal integrator means 367and the short signal integrator means 368 for each block integrate theexposure amount of each pixel for each block divided into blocks in theshort signal (short frame signal) and the long signal (long framesignal) read from the image memory 34, respectively, to determine thetotal of exposure amount (SSUM, LSUM) in block units. During period{circle around (3)}, the timing of reading the short signal and the longsignal from the image memory 34 is such that the lines on the respectivepositions on the solid-state imaging element 32 are read sequentiallyfrom the first line at the same timing.

The short signal and the long signal inputted in synchronism with theline into the exposure amount ratio detector means 36 are determined,respectively, for the level in the signal level determination means 361,and a signal determined to be at a level out of a predetermined range isneglected so as not to be integrated for each block in the long signalintegrator means 367 and the short signal integrator means 368. Here,the level out of a predetermined range means that it is a level at whichthe ratio in the exposure amount is difficult to determine, and morespecifically, that the long signal is a signal of a high brightnessportion near the saturation level of the solid-state imaging element 32,while the short signal is a signal of a low brightness portion on whichthe noise effect largely acts.

A signal determined to be at a level within a predetermined range isswitched in the route by the multiplexers 365 a, 365 b and integrated bythe integrator means ΣBL1, ΣBL2 to ΣBLn in block units so as to beintegrated in blocks to which each pixel signal corresponds.

During period {circle around (4)}, when the long signal and the shortsignal of the pixel in nth (first to 48th) block of those divided intoblocks are outputted from the image memory 34, the selector 366 iscontrolled in a manner that a signal used to determine the integratedvalue of the exposure amount of nth block is inputted into the exposureamount ratio operation means 364, which in turn operates the ratio ofthe long signal to the short signal in nth block, and then a gainequivalent to the exposure amount ratio is given by the gain adjustmentmeans 37 to the short signal from the image memory 34. A signal afterthe gain is given by the gain adjustment means 37 to the short signal isexpressed as the short′ signal hereinafter. During period {circle around(4)}, based on the long signal read from the image memory 34 and on theshort′ signal whose gain is adjusted, a synthesization is performed bythe image synthesizer means 38 at a post-stage with a method as shown inthe well-known principle of the dynamic range expansion.

With the above-mentioned method, even a subject on which a plurality oflight sources exist, the ratio in the exposure amount of the long signalto the short signal in block units can be determined at a high accuracy,so that the synthesis image by the image synthesizer means 38 using thelong signal and the gain-adjusted short signal can obtain gradationcharacteristics which are continuously stable from a low-brightnessportion to a high-brightness portion.

The signal synthesized in the image synthesizer means 38 is subject toprocessings such as separating of brightness signal from color signal,noise removing, edge enhancing, gamma correcting, matrix operating, andencoding to a specific format, in the later processing, as withEmbodiment 5.

Although in Embodiment 7, a case where the exposure amount ratio isdetermined for each block obtained by dividing the screen into 48blocks, the present invention is not limited to such a case, and thescreen may be divided into any number of blocks. It is sufficient thatthe number of the integrator means for each block in the long signalintegrator means 367 and the short signal integrator means 368 isdecided so as to be matched with the number of divided blocks. InEmbodiment 7, the block may be divided in the horizontal/verticaldirection at equal intervals, or the interval in the horizontal/verticaldirection may be freely changed according to a subject.

(7) Embodiment 8

The brightness of a fluorescent lamp, as shown in a graph of thefluctuation in brightness for each color component of a fluorescent lampof FIG. 65, varies in the phase for each color component, so that theexposure amount ratio varies for each color component with the timing ofperforming exposure. In a method of using a plurality of image signalshaving a different exposure amount and synthesizing the signals to pickup an image with an expanded dynamic range, a synthesis signal obtainedby determining the exposure amount ratio in only brightness portion ofthe long-time exposure signal and the short-time exposure signal, and byperforming the image synthesis to expand the dynamic range with both thelong-time exposure signal and a signal giving a gain equivalent to theexposure amount ratio to the short-time exposure signal has colorcomponent characteristics which become discontinuous near a position atwhich the long-time exposure signal saturates.

The eighth embodiment intends to provide a synthesis image having stablycontinuous gradation characteristics and color characteristics after theimage synthesis processing for expanding the dynamic range even when thephase is different for each color component, by determining the exposureamount ratio of the short-time exposure signal (short signal) to thelong-time exposure signal (long signal) for each color component.

FIG. 63 is a block diagram showing a configuration of the solid-stateimaging device in Embodiment 8 of the present invention. In FIG. 63, theconfiguration is exactly the same as Embodiment 5 other than theinternal configuration of the exposure amount ratio detector means 36,so that the explanation will be omitted here. In the exposure amountratio detector means 36 of FIG. 61, the numeral 361 designates signallevel determination means; the numerals 365 a, 365 b, multiplexers; thenumerals 366 a, 366 b, selectors; the numeral 369, a long signalintegrator means for determining an integrated value LSUM of the longsignal for each color component; and the numeral 3610, a short signalintegrator means for determining an integrated value SSUM of the shortsignal for each color component. In the long signal integrator means 369and the short signal integrator means 3610 for each color component, thesymbols ΣMY, ΣMC, ΣGY and ΣGC designate integrator means for each mixedcolor component of [magenta (Mg)+yellow (Ye)], [magenta (Mg)+cyan (Cy)],[green (G)+yellow (Ye)] and [green (G)+cyan (Cy)], respectively. Thenumeral 364 designates exposure amount ratio operation means forreceiving the integrated value LSUM for each color component by the longsignal integrator means 369 and the integrated value SSUM for each colorcomponent by the short signal integrator means 3610 obtained byswitching between these integrated values by the selectors 366 a, 366 bto determine the ratio of the integrated value LSUM to the integratedvalue SSUM for each color component.

With respect to the solid-state imaging device of the present eighthembodiment configured as described above, the operation thereof will beexplained hereinafter.

Here, the color filters on the photodiode of the solid-state imagingelement 32 are assumed to be arranged in a manner that the color filtershaving four different spectral characteristics of magenta (Mg), green(G), yellow (Ye) and cyan (Cy) are arranged for each pixel.

First, with respect to the method of picking up the short signal and thelong signal and to the method of controlling the exposure time and thesolid-state imaging element 32 and the image memory 34, the concreteexample in Embodiment 8 is exactly the same as that described in fifthembodiment, so that the explanation will be omitted here. The timing ofthe output signal from the solid-state imaging element 32 and the timingof the input/output signal into/from the image memory 34 at that timeare exactly the same as the timing chart of FIG. 57 used to explainEmbodiment 5, so that Embodiment 8 will be explained also using FIG. 57.

In Embodiment 8, during periods {circle around (1)}, {circle around (2)}shown in FIG. 57, the signal outputted from the solid-state imagingelement 32 is subject to the two horizontal lines addition processing bythe two horizontal line adder means 33, and then is stored for a time inthe image memory 34.

During period {circle around (3)}, in the respective long signal and therespective short signal read from the image memory 34, the exposureamount of each pixel for each color component is integrated to determinethe total of exposure amount (SSUM, LSUM) for each color component.

During period {circle around (3)}, the timing of reading the shortsignal and the long signal from the image memory 34 is such that thelines on the respective positions on the solid-state imaging element 32are read sequentially from the first line at the same timing.

The short signal and the long signal inputted in synchronism with theline into the exposure amount ratio detector means 36 are determined,respectively, for the level in the signal level determination means 361,and a signal determined to be at a level out of a predetermined range isneglected so as not to be integrated for each color component in thelong signal integrator means 369 and the short signal integrator means3610. Here, the level out of a predetermined range means that it is alevel at which the ratio in the exposure amount is difficult todetermine, and more specifically, that the long signal is a signal of ahigh brightness portion near the saturation level of the solid-stateimaging element 32, while the short signal is a signal of a lowbrightness portion on which the noise effect largely acts.

A signal determined to be at a level within a predetermined range isswitched in the route by the multiplexers 365 a, 365 b and integrated bythe integrator means ΣMY, ΣMC, ΣGY and ΣGC for each color component soas to be integrated for each color component to which each pixel signalcorresponds.

The signals outputted from the image memory 34 are mixed for two linesin the two horizontal line adder means 33, so that the signals of fourkinds of [magenta (Mg)+yellow (Ye)], [magenta (Mg)+cyan (Cy)], [green(G)+yellow (Ye)] and [green (G)+cyan (Cy)] as color components areoutputted, and thus integrated for each component of the colors of thefour kinds.

During period {circle around (4)}, when the long signal and the shortsignal of the color component of [magenta (Mg)+yellow (Ye)] areoutputted from the image memory 34, the selectors 366 a, 366 b arecontrolled in a manner that a signal used to determine the integratedvalue of the exposure amount of [magenta (Mg)+yellow (Ye)] is inputtedinto the exposure amount ratio operation means 364, which in turnoperates the ratio of the long signal to the short signal in [magenta(Mg)+yellow (Ye)], and then a gain equivalent to the exposure amountratio is given by the gain adjustment means 37 to the short signal ofthe color component of [magenta (Mg)+yellow (Ye)] from the image memory34,

Similarly, during period {circle around (4)}, when the long signal andthe short signal of the color component of [magenta (Mg)+cyan (Cy)],[green (G)+yellow (Ye)] and [green (G)+cyan (Cy)] are outputted from theimage memory 34, the same processing is performed. A signal after thegain is given by the gain adjustment means 37 to the short signal isexpressed as the short′ signal hereinafter.

Based on the long signal and on the short′ signal whose gain isadjusted, a synthesization is performed by the image synthesizer means38 at a post-stage with a method as shown in the conventional principleof the dynamic range expansion.

With the above-mentioned method, even when as with a fluorescent lamp,the phase is different for each color component, the ratio in theexposure amount of the long signal to the short signal over the wholescreen can be determined at a high accuracy, so that the synthesis imageby the image synthesizer means 38 using the long signal and thegain-adjusted short signal can obtain gradation characteristics andcolor characteristics which are continuously stable from alow-brightness portion to a high-brightness portion.

The signal synthesized in the image synthesizer means 38 is subject toprocessings such as separating of brightness signal from color signal,noise removing, edge enhancing, gamma correcting, matrix operating, andencoding to a specific format, in the later processing, as withEmbodiment 5.

Although in Embodiment 8, the arrangement of color filter for magenta,cyan, green, yellow on the photodiode of the solid-state imaging element32 is explained, the present invention is not to limited to such anarrangement, and can be applied also to the arrangement of colorseparation filters with various and different spectral characteristics.The present invention can be applied also to the arrangement ofcolorization means by color separation prisms in place of colorseparation filters.

There can be an embodiment obtained by combining Embodiment 7 withEmbodiment 8, though not shown. That is, it is sufficient that theconfiguration is such that the integrator means ΣMY, ΣMC, ΣGY and ΣGCfor each color component are added to respective the integrator meansΣBL1, ΣBL2 through Z BLn in block units.

INDUSTRIAL APPLICABILITY

The present invention is utilized suitably as a solid-state imagingdevice which controls the exposure of a solid-state imaging element andthe signal read mode, picks up the short-time exposure signal for onefield and the long-time exposure signal for one frame, and synthesizesthese signals, thereby allowing an image whose dynamic range is expandedwhile having a resolution equivalent to the number of pixels of thesolid-state imaging element to be picked up.

1-42. (canceled)
 43. A solid-state imaging device for performing adynamic range expansion by synthesizing a long-time exposure signal anda short-time exposure signal obtained by imaging, said solid-stateimaging device detecting an exposure amount ratio of said long-timeexposure signal to said short-time exposure signal, and performing again adjustment of said long-time exposure signal and said short-timeexposure signal-in said dynamic range expansion in accordance with saidexposure amount ratio.
 44. A solid-state imaging device according toclaim 43, wherein a detection of said exposure amount ratio of saidlong-time exposure signal to said short-time exposure signal isperformed for each block divided and formed on an imaging screen, andsaid gain adjustment is performed for each said block.
 45. A solid-stateimaging device according to claim 43, wherein a detection of saidexposure amount ratio of said long-time exposure signal to saidshort-time exposure signal is performed for each color component, andsaid gain adjustment is performed for each said color component.
 46. Asolid-state imaging device according to claim 43, wherein a detection ofsaid exposure amount ratio of said long-time exposure signal to saidshort-time exposure signal is performed for each block divided andformed on an imaging screen and for each color component, and said gainadjustment is performed for each said block and for each said colorcomponent.
 47. A solid-state imaging device comprising: imaging meansfor outputting independently a plurality of image signals having adifferent exposure amount; an image memory for storing said plurality ofimage signals having a different exposure amount from said imagingmeans, an exposure amount ratio detector means for detecting an exposureamount ratio of said plurality of image signals having a differentexposure mount; gain adjustment means for performing a gain adjustmentto said plurality of image signals having a different exposure amountread from said image memory based on said detected exposure amountratio; and image synthesizer means for synthesizing said plurality ofimage signals having a different exposure amount after said gainadjustment.
 48. A solid-state imaging device according to claim 47,wherein said exposure amount ratio detector means includes: signal leveldetermination means for determining a signal level of said plurality ofimage signals having a different exposure amount, integrator means fordetermining an integrated value for each of said plurality of imagesignals having a different exposure amount within a predetermined rangeby a determination on said signal level, and exposure amount ratiooperation means for determining a ratio of their plural integratedvalues.
 49. A solid-state imaging device according to claim 48, whereinsaid integrator means is configured in such a manner as to determinesaid integrated values for each of said plurality of image signalshaving a different exposure amount in units of the block obtained bydividing the screen into a plurality of blocks.
 50. A solid-stateimaging device according to claim 48, wherein said imaging means hascolorization means by the use of a color separation filer or a colorseparation prism having different spectral characteristics, and further,said integrator means is configured in a manner to determine anintegrated value for each color component and for each of the pluralityof image signals having a different exposure amount.
 51. A solid-stateimaging device according to claim 48, wherein said imaging means hascolorization means by the use of a color separation filer or a colorseparation prism having different spectral characteristics, and further,said integrator means is configured in a manner to determine anintegrated value for each color component and for each of the pluralityof image signals having a different exposure amount and in units of theblock obtained by dividing the screen into a plurality of blocks.
 52. Asolid-state imaging device according to claim 47, wherein there isincluded image signal switching means for switching between said imagesignal from said imaging means and said image signal from said imagememory for one of the plurality of image signals having a differentexposure amount which are outputted from said imaging means to saidimage memory.
 53. A solid-state imaging device according to claim 47,wherein said gain adjustment means to the plurality of image signalshaving a different exposure amount is configured in a manner to give again based on said exposure amount ratio detected by said exposureamount ratio detector means to at least one of said plurality of imagesignals having a different exposure amount.